[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:02:56 dist-test-slave-zcsk krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:02:56 dist-test-slave-zcsk krb5kdc[399](info): set up 2 sockets
Sep 24 04:02:56 dist-test-slave-zcsk krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 04:02:57.763794 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:02:57 dist-test-slave-zcsk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:02:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:02:57.881028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43149
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:02:57.987562 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:02:57.987838 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:02:57.987876 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:02:57.991859 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:02:57.991969 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:02:57.991999 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:02:57.992030 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:02:57.992060 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:02:57.997510 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:02:57.998579 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:02:57.999660 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:02:58.008157 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:02:58.008143 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:02:58.008203 412 server_base.cc:1034] running on GCE node
W20260924 04:02:58.008143 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:02:58.008996 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:02:58.010089 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:02:58.011247 412 hybrid_clock.cc:648] HybridClock initialized: now 1790222578011213 us; error 46 us; skew 500 ppm
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:02:58.014473 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:02:58.015534 412 webserver.cc:460] Webserver started at http://127.0.98.62:46851/ using document root <none> and password file <none>
I20260924 04:02:58.016260 412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:02:58.016350 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:02:58.016623 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:02:58.019408 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "449a32de98634a74aaea55b739ba520a"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "eea344eb9d7345d549f9e8261d177928"
server_key_iv: "0455c4f98f3e057e1b06cf4ac1780690"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.020183 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "449a32de98634a74aaea55b739ba520a"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "eea344eb9d7345d549f9e8261d177928"
server_key_iv: "0455c4f98f3e057e1b06cf4ac1780690"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.024346 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:02:58.026908 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.028277 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:02:58.028400 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "449a32de98634a74aaea55b739ba520a"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "eea344eb9d7345d549f9e8261d177928"
server_key_iv: "0455c4f98f3e057e1b06cf4ac1780690"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.028512 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:02:58.040280 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:02:58.043098 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:02:58.043278 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:02:58.049930 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43149
I20260924 04:02:58.049947 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43149 every 8 connection(s)
I20260924 04:02:58.050850 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:02:58.053659 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:02:58.056936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 412
I20260924 04:02:58.057060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:02:58.058274 392 external_mini_cluster.cc:1428] Setting key c4896ec1b7596fff63d3c20c373d5302
I20260924 04:02:58.059052 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a: Bootstrap starting.
I20260924 04:02:58.061695 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a: Neither blocks nor log segments found. Creating new log.
I20260924 04:02:58.062577 480 log.cc:824] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:02:58.064729 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a: No bootstrap required, opened a new log
I20260924 04:02:58.069427 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449a32de98634a74aaea55b739ba520a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } }
I20260924 04:02:58.069692 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:02:58.069751 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449a32de98634a74aaea55b739ba520a, State: Initialized, Role: FOLLOWER
I20260924 04:02:58.070222 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449a32de98634a74aaea55b739ba520a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } }
I20260924 04:02:58.070379 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:02:58.070454 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:02:58.070549 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:02:58.071382 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449a32de98634a74aaea55b739ba520a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } }
I20260924 04:02:58.071686 480 leader_election.cc:302] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 449a32de98634a74aaea55b739ba520a; no voters:
I20260924 04:02:58.071995 480 leader_election.cc:288] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:02:58.072067 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:02:58.072292 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [term 1 LEADER]: Becoming Leader. State: Replica: 449a32de98634a74aaea55b739ba520a, State: Running, Role: LEADER
I20260924 04:02:58.072625 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449a32de98634a74aaea55b739ba520a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } }
I20260924 04:02:58.073273 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:02:58.074731 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "449a32de98634a74aaea55b739ba520a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449a32de98634a74aaea55b739ba520a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } }
I20260924 04:02:58.074849 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [sys.catalog]: This master's current role is: LEADER
I20260924 04:02:58.075372 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 449a32de98634a74aaea55b739ba520a. Latest consensus state: current_term: 1 leader_uuid: "449a32de98634a74aaea55b739ba520a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449a32de98634a74aaea55b739ba520a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } }
I20260924 04:02:58.075552 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a [sys.catalog]: This master's current role is: LEADER
I20260924 04:02:58.075559 497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:02:58.078632 497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:02:58.084337 497 catalog_manager.cc:1347] Generated new cluster ID: 7b07e838c3db458e864badf601493ecb
I20260924 04:02:58.084409 497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:02:58.085059 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.061141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.0.98.62:43149)
0924 04:02:58.061648 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 04:02:58.061669 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:02:58.062832 (+ 1163us) server_negotiation.cc:407] Connection header received
0924 04:02:58.067158 (+ 4326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:02:58.067175 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:02:58.067518 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:02:58.067780 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:02:58.070649 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.071703 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.073383 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.073964 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.074119 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.078804 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:02:58.078811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:02:58.078826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:02:58.078869 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:02:58.080943 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.081553 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.081559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.081565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.081700 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.082075 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.082078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.082079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.082558 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:02:58.082741 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 04:02:58.083466 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 04:02:58.083760 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":336,"thread_start_us":148,"threads_started":1}
I20260924 04:02:58.097225 497 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:02:58.098152 497 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:02:58.101542 497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 449a32de98634a74aaea55b739ba520a: Generated new TSK 0
I20260924 04:02:58.102171 497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:02:58.129045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43149
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:02:58.238775 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:02:58.239068 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:02:58.239113 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:02:58.243156 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:02:58.243222 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:02:58.243283 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:02:58.248589 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:02:58.249590 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:02:58.250541 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:02:58.260181 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:02:58.260190 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:02:58.260195 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:02:58.260421 506 server_base.cc:1034] running on GCE node
I20260924 04:02:58.260852 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:02:58.261521 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:02:58.262691 506 hybrid_clock.cc:648] HybridClock initialized: now 1790222578262660 us; error 41 us; skew 500 ppm
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:02:58.265377 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:02:58.266248 506 webserver.cc:460] Webserver started at http://127.0.98.1:45101/ using document root <none> and password file <none>
I20260924 04:02:58.266789 506 fs_manager.cc:359] Metadata directory not provided
I20260924 04:02:58.266863 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:02:58.267068 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:02:58.269145 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "2135851760074236857bcdfead051451"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "b8d46b33a3751794e9bd4031e74798bd"
server_key_iv: "14cf2c7ac8288c594ed6ef0414a63889"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.269734 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2135851760074236857bcdfead051451"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "b8d46b33a3751794e9bd4031e74798bd"
server_key_iv: "14cf2c7ac8288c594ed6ef0414a63889"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.273412 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:02:58.276103 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.277554 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:02:58.277704 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "2135851760074236857bcdfead051451"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "b8d46b33a3751794e9bd4031e74798bd"
server_key_iv: "14cf2c7ac8288c594ed6ef0414a63889"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.277848 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:02:58.297857 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:02:58.301252 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:02:58.301433 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:02:58.302002 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:02:58.302804 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:02:58.302850 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.302902 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:02:58.302928 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.311697 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43005
I20260924 04:02:58.311718 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43005 every 8 connection(s)
I20260924 04:02:58.312597 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:02:58.314837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 506
I20260924 04:02:58.314918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:02:58.315161 392 external_mini_cluster.cc:1428] Setting key 92fe4119895f3dbec3976a1bcd6db297
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:02:58.327525 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.314366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58091 (local address 127.0.98.62:43149)
0924 04:02:58.314542 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 04:02:58.314548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:02:58.315464 (+ 916us) server_negotiation.cc:407] Connection header received
0924 04:02:58.316874 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:02:58.316877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:02:58.316924 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:02:58.316997 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:02:58.318462 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.319279 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.320892 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.321142 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.321222 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.323965 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:02:58.323967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:02:58.323969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:02:58.323992 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:02:58.325318 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.325861 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.325864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.325865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.325945 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.326308 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.326311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.326327 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.326500 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:02:58.326601 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:02:58.327197 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 04:02:58.327361 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:02:58.328344 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.314799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43149 (local address 127.0.98.1:58091)
0924 04:02:58.315291 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 04:02:58.315358 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 04:02:58.316636 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:02:58.317182 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:02:58.317192 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:02:58.317647 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:02:58.318313 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:02:58.318334 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.319419 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.319422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:02:58.320766 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:02:58.320777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.321283 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.321287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:02:58.321546 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.322109 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:02:58.322131 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:02:58.323782 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:02:58.325430 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:02:58.325435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:02:58.325462 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:02:58.325747 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:02:58.326058 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:02:58.326061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:02:58.326063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:02:58.326194 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:02:58.326615 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:02:58.326626 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:02:58.326927 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 04:02:58.327131 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 04:02:58.327465 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260924 04:02:58.329576 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43149
I20260924 04:02:58.329799 635 heartbeater.cc:464] Registering TS with master...
I20260924 04:02:58.330262 635 heartbeater.cc:511] Master 127.0.98.62:43149 requested a full tablet report, sending...
I20260924 04:02:58.332034 444 ts_manager.cc:194] Registered new tserver with Master: 2135851760074236857bcdfead051451 (127.0.98.1:43005)
I20260924 04:02:58.333675 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:02:58.343554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43149
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:02:58.449095 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:02:58.449353 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:02:58.449393 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:02:58.453336 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:02:58.453393 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:02:58.453459 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:02:58.458678 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:02:58.459637 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:02:58.460621 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:02:58.470198 640 server_base.cc:1034] running on GCE node
W20260924 04:02:58.470145 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:02:58.470232 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:02:58.470335 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:02:58.470831 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:02:58.471482 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:02:58.472663 640 hybrid_clock.cc:648] HybridClock initialized: now 1790222578472649 us; error 28 us; skew 500 ppm
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:02:58.475616 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:02:58.476622 640 webserver.cc:460] Webserver started at http://127.0.98.2:39855/ using document root <none> and password file <none>
I20260924 04:02:58.477185 640 fs_manager.cc:359] Metadata directory not provided
I20260924 04:02:58.477268 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:02:58.477481 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:02:58.479650 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "0b876db3df4c4326819c97718ee725cb"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "af42ba500936cddc65f730069c15db47"
server_key_iv: "cfa3eed0bb74c1de4e723f50d48ed797"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.480288 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0b876db3df4c4326819c97718ee725cb"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "af42ba500936cddc65f730069c15db47"
server_key_iv: "cfa3eed0bb74c1de4e723f50d48ed797"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.484398 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:02:58.486963 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.488399 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:02:58.488513 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "0b876db3df4c4326819c97718ee725cb"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "af42ba500936cddc65f730069c15db47"
server_key_iv: "cfa3eed0bb74c1de4e723f50d48ed797"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.488633 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:02:58.503273 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:02:58.506403 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:02:58.506559 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:02:58.507088 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:02:58.507876 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:02:58.507947 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.507992 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:02:58.508005 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.517680 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41933
I20260924 04:02:58.517704 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41933 every 8 connection(s)
I20260924 04:02:58.518589 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:02:58.519362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 640
I20260924 04:02:58.519475 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:02:58.519737 392 external_mini_cluster.cc:1428] Setting key 8568907a231ce7f64fdd1a2cb63ff16d
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:02:58.535408 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.520559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60677 (local address 127.0.98.62:43149)
0924 04:02:58.520685 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:02:58.520689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:02:58.521697 (+ 1008us) server_negotiation.cc:407] Connection header received
0924 04:02:58.523503 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:02:58.523507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:02:58.523565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:02:58.523650 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:02:58.525038 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.525559 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.526759 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.526985 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.527072 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.531817 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:02:58.531820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:02:58.531822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:02:58.531856 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:02:58.533471 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.533954 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.533957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.533959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.534037 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.534352 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.534355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.534357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.534540 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:02:58.534624 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:02:58.535118 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 04:02:58.535240 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:02:58.536289 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.520986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43149 (local address 127.0.98.2:60677)
0924 04:02:58.521527 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 04:02:58.521587 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:02:58.523162 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:02:58.523822 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:02:58.523830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:02:58.524326 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:02:58.524890 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:02:58.524905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.525677 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.525680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:02:58.526645 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:02:58.526656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.527125 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.527129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:02:58.527289 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.527833 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:02:58.527851 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:02:58.531638 (+ 3787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:02:58.533580 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:02:58.533586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:02:58.533604 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:02:58.533858 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:02:58.534125 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:02:58.534128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:02:58.534130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:02:58.534268 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:02:58.534633 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:02:58.534641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:02:58.534892 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 04:02:58.535060 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:02:58.535410 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":149,"threads_started":1}
I20260924 04:02:58.537592 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43149
I20260924 04:02:58.537812 769 heartbeater.cc:464] Registering TS with master...
I20260924 04:02:58.538304 769 heartbeater.cc:511] Master 127.0.98.62:43149 requested a full tablet report, sending...
I20260924 04:02:58.539265 444 ts_manager.cc:194] Registered new tserver with Master: 0b876db3df4c4326819c97718ee725cb (127.0.98.2:41933)
I20260924 04:02:58.539932 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:02:58.547231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43149
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:02:58.659543 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:02:58.659818 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:02:58.659864 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:02:58.663937 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:02:58.664005 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:02:58.664065 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:02:58.669409 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:02:58.670455 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:02:58.671482 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:02:58.681087 774 server_base.cc:1034] running on GCE node
W20260924 04:02:58.681012 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:02:58.681006 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:02:58.681031 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:02:58.681699 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:02:58.682348 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:02:58.683507 774 hybrid_clock.cc:648] HybridClock initialized: now 1790222578683487 us; error 35 us; skew 500 ppm
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:02:58.686084 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:02:58.686946 774 webserver.cc:460] Webserver started at http://127.0.98.3:39981/ using document root <none> and password file <none>
I20260924 04:02:58.687448 774 fs_manager.cc:359] Metadata directory not provided
I20260924 04:02:58.687502 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:02:58.687656 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:02:58.689641 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbfc8818e6524908a32a3e60ee69722e"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "b2213941b447df9fc8f2b54e8c3a5640"
server_key_iv: "740910e1d07d514b49da36eab5427f3e"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.690142 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbfc8818e6524908a32a3e60ee69722e"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "b2213941b447df9fc8f2b54e8c3a5640"
server_key_iv: "740910e1d07d514b49da36eab5427f3e"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.693856 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:02:58.696295 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.697710 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:02:58.697824 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "fbfc8818e6524908a32a3e60ee69722e"
format_stamp: "Formatted at 2026-09-24 04:02:58 on dist-test-slave-zcsk"
server_key: "b2213941b447df9fc8f2b54e8c3a5640"
server_key_iv: "740910e1d07d514b49da36eab5427f3e"
server_key_version: "encryptionkey@0"
I20260924 04:02:58.697945 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:02:58.720294 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:02:58.723405 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:02:58.723579 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:02:58.724205 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:02:58.725035 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:02:58.725085 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.725124 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:02:58.725150 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:02:58.734781 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38507
I20260924 04:02:58.734812 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38507 every 8 connection(s)
I20260924 04:02:58.735702 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:02:58.743220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 774
I20260924 04:02:58.743345 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:02:58.743614 392 external_mini_cluster.cc:1428] Setting key 980b136b9e6df5b5e2d89f64a6107c6a
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:02:58.749250 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.737570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44589 (local address 127.0.98.62:43149)
0924 04:02:58.737720 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:02:58.737724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:02:58.738667 (+ 943us) server_negotiation.cc:407] Connection header received
0924 04:02:58.739977 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:02:58.739980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:02:58.740029 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:02:58.740105 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:02:58.741285 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.741809 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.742928 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.743134 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.743201 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.745894 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:02:58.745898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:02:58.745901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:02:58.745930 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:02:58.747287 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.747756 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.747768 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.747770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.747860 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.748190 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.748193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.748195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.748379 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:02:58.748463 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:02:58.748962 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 04:02:58.749090 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:02:58.750072 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.738041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43149 (local address 127.0.98.3:44589)
0924 04:02:58.738506 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 04:02:58.738550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:02:58.739728 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:02:58.740267 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:02:58.740275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:02:58.740636 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:02:58.741144 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:02:58.741157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.741923 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.741926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:02:58.742824 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:02:58.742832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.743274 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.743279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:02:58.743477 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.744100 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:02:58.744122 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:02:58.745741 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:02:58.747394 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:02:58.747399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:02:58.747417 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:02:58.747666 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:02:58.747957 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:02:58.747960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:02:58.747962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:02:58.748106 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:02:58.748473 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:02:58.748480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:02:58.748735 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 04:02:58.748911 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 04:02:58.749242 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260924 04:02:58.751175 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43149
I20260924 04:02:58.751472 903 heartbeater.cc:464] Registering TS with master...
I20260924 04:02:58.752014 903 heartbeater.cc:511] Master 127.0.98.62:43149 requested a full tablet report, sending...
I20260924 04:02:58.753069 444 ts_manager.cc:194] Registered new tserver with Master: fbfc8818e6524908a32a3e60ee69722e (127.0.98.3:38507)
I20260924 04:02:58.753655 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44589
I20260924 04:02:58.758359 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:02:58.784279 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.773807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.0.98.1:43005)
0924 04:02:58.774147 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 04:02:58.774160 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:02:58.774189 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:02:58.774246 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:02:58.774253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:02:58.774489 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:02:58.774623 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:02:58.775322 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.776209 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.777064 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.777451 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.777548 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.780493 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:02:58.780503 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:02:58.780518 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:02:58.780549 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:02:58.782320 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.782829 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.782836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.782837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.782934 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.783205 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.783207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.783209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.783507 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:02:58.783679 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 04:02:58.783966 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 04:02:58.784116 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":133,"threads_started":1}
W20260924 04:02:58.785293 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44556
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:02:58.811084 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:02:58.802449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.0.98.1:43005)
0924 04:02:58.802576 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:02:58.802580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:02:58.802695 (+ 115us) server_negotiation.cc:407] Connection header received
0924 04:02:58.802833 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:02:58.802835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:02:58.802877 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:02:58.802994 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:02:58.803729 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.804352 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.805172 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:02:58.805421 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:02:58.805521 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:02:58.807946 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:02:58.807950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:02:58.807954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:02:58.807986 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:02:58.809445 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.809838 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.809840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.809842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.809927 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:02:58.810187 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:02:58.810189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:02:58.810190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:02:58.810411 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:02:58.810519 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:02:58.810790 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 04:02:58.810921 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:02:58.812074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 506
I20260924 04:02:58.817215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 640
I20260924 04:02:58.823786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 774
I20260924 04:02:58.828603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 412
2026-09-24T04:02:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2034 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[917](info): set up 2 sockets
Sep 24 04:02:58 dist-test-slave-zcsk krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 04:02:59.783444 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:02:59 dist-test-slave-zcsk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:02:59.899827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:00.008829 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:00.009152 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:00.009223 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:00.013793 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:00.013895 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:00.013928 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:00.013971 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:00.014007 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:00.021642 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:00.023216 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:00.024505 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:00.033566 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:00.033557 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:00.033648 930 server_base.cc:1034] running on GCE node
W20260924 04:03:00.033557 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:00.034375 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:00.035501 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:00.036689 930 hybrid_clock.cc:648] HybridClock initialized: now 1790222580036640 us; error 71 us; skew 500 ppm
Sep 24 04:03:00 dist-test-slave-zcsk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:00.039655 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:00.040673 930 webserver.cc:460] Webserver started at http://127.0.98.62:45239/ using document root <none> and password file <none>
I20260924 04:03:00.041209 930 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:00.041267 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:00.041440 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:00.043658 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "b1df6c0b3be34a0489a7097927f71bea"
format_stamp: "Formatted at 2026-09-24 04:03:00 on dist-test-slave-zcsk"
server_key: "76ebb947f5f3501df291bd2ca06bdae0"
server_key_iv: "0240dd12fc14bfa1d7b918e0090df1b4"
server_key_version: "encryptionkey@0"
I20260924 04:03:00.044219 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1df6c0b3be34a0489a7097927f71bea"
format_stamp: "Formatted at 2026-09-24 04:03:00 on dist-test-slave-zcsk"
server_key: "76ebb947f5f3501df291bd2ca06bdae0"
server_key_iv: "0240dd12fc14bfa1d7b918e0090df1b4"
server_key_version: "encryptionkey@0"
I20260924 04:03:00.048180 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:03:00.050668 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:00.052021 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:00.052126 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "b1df6c0b3be34a0489a7097927f71bea"
format_stamp: "Formatted at 2026-09-24 04:03:00 on dist-test-slave-zcsk"
server_key: "76ebb947f5f3501df291bd2ca06bdae0"
server_key_iv: "0240dd12fc14bfa1d7b918e0090df1b4"
server_key_version: "encryptionkey@0"
I20260924 04:03:00.052217 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:00.062572 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:00.065580 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:00.065732 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:00.073035 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46493
I20260924 04:03:00.073087 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46493 every 8 connection(s)
I20260924 04:03:00.074297 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:00.075698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 930
I20260924 04:03:00.075803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:00.076089 392 external_mini_cluster.cc:1428] Setting key 5cc1936ddfd97a37d8bb97068a41f0ca
I20260924 04:03:00.078944 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:00.085832 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea: Bootstrap starting.
Sep 24 04:03:00 dist-test-slave-zcsk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:00.088390 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:00.089120 998 log.cc:824] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:00.091322 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea: No bootstrap required, opened a new log
I20260924 04:03:00.092089 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:00.077887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.98.62:46493)
0924 04:03:00.078329 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:00.078347 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:00.078393 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:03:00.079234 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:00.079257 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:00.079626 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:00.079988 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:00.080877 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:00.082424 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:00.083627 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:00.084413 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:00.084573 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:00.087453 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:00.087461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:00.087479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:00.087513 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:00.089123 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:00.089592 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:00.089598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:00.089604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:00.089740 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:00.090066 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:00.090069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:00.090070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:00.090489 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:00.090642 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:00.090924 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 04:03:00.091277 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":140,"threads_started":1}
I20260924 04:03:00.094861 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df6c0b3be34a0489a7097927f71bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46493 } }
I20260924 04:03:00.095079 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:00.095134 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1df6c0b3be34a0489a7097927f71bea, State: Initialized, Role: FOLLOWER
I20260924 04:03:00.095538 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df6c0b3be34a0489a7097927f71bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46493 } }
I20260924 04:03:00.095693 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:00.095759 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:00.095860 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:00.096920 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df6c0b3be34a0489a7097927f71bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46493 } }
I20260924 04:03:00.097236 998 leader_election.cc:302] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1df6c0b3be34a0489a7097927f71bea; no voters:
I20260924 04:03:00.097515 998 leader_election.cc:288] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:00.097675 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:00.097869 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [term 1 LEADER]: Becoming Leader. State: Replica: b1df6c0b3be34a0489a7097927f71bea, State: Running, Role: LEADER
I20260924 04:03:00.098150 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df6c0b3be34a0489a7097927f71bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46493 } }
I20260924 04:03:00.098673 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:00.099715 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1df6c0b3be34a0489a7097927f71bea. Latest consensus state: current_term: 1 leader_uuid: "b1df6c0b3be34a0489a7097927f71bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df6c0b3be34a0489a7097927f71bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46493 } } }
I20260924 04:03:00.099838 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:00.100225 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1df6c0b3be34a0489a7097927f71bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1df6c0b3be34a0489a7097927f71bea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46493 } } }
I20260924 04:03:00.100296 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:00.100323 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:00.103431 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:00.108659 1012 catalog_manager.cc:1347] Generated new cluster ID: 9a5912dd946a47b2ae8537b2e166fa1f
I20260924 04:03:00.108731 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:00.127277 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:00.128273 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:00.134955 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1df6c0b3be34a0489a7097927f71bea: Generated new TSK 0
I20260924 04:03:00.135650 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:00.175294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46493
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:00.298264 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:00.298538 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:00.298583 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:00.302790 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:00.302870 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:00.302935 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:00.308230 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:00.309257 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:00.310274 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:00.320003 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:00.320046 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:00.320019 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:00.320072 1024 server_base.cc:1034] running on GCE node
I20260924 04:03:00.320567 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:00.321206 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:00.322399 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790222580322368 us; error 45 us; skew 500 ppm
Sep 24 04:03:00 dist-test-slave-zcsk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:00.325182 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:00.326061 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45977/ using document root <none> and password file <none>
I20260924 04:03:00.326599 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:00.326664 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:00.326858 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:00.328969 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4e9a77aa8644abd993fc8317bc56fb2"
format_stamp: "Formatted at 2026-09-24 04:03:00 on dist-test-slave-zcsk"
server_key: "b7b3c107ed81f2d2d81def1e7ce68498"
server_key_iv: "247fa1fb322e3322426b85c32649e615"
server_key_version: "encryptionkey@0"
I20260924 04:03:00.329545 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4e9a77aa8644abd993fc8317bc56fb2"
format_stamp: "Formatted at 2026-09-24 04:03:00 on dist-test-slave-zcsk"
server_key: "b7b3c107ed81f2d2d81def1e7ce68498"
server_key_iv: "247fa1fb322e3322426b85c32649e615"
server_key_version: "encryptionkey@0"
I20260924 04:03:00.333494 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:03:00.337040 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:00.338878 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:00.339038 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "d4e9a77aa8644abd993fc8317bc56fb2"
format_stamp: "Formatted at 2026-09-24 04:03:00 on dist-test-slave-zcsk"
server_key: "b7b3c107ed81f2d2d81def1e7ce68498"
server_key_iv: "247fa1fb322e3322426b85c32649e615"
server_key_version: "encryptionkey@0"
I20260924 04:03:00.339181 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:00.356786 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:00.359884 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:00.360107 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:00.360700 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:00.361582 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:00.361657 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:00.361756 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:00.361807 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:00.370332 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37915
I20260924 04:03:00.370368 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37915 every 8 connection(s)
I20260924 04:03:00.371269 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:00.372113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 1024
I20260924 04:03:00.372210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:00.372493 392 external_mini_cluster.cc:1428] Setting key 9d99eb2dc7abd8f8f237c53456ccaeb2
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 24 04:03:00 dist-test-slave-zcsk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:00.549984 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:00.373249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57859 (local address 127.0.98.62:46493)
0924 04:03:00.373381 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:00.373385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:00.374346 (+ 961us) server_negotiation.cc:407] Connection header received
0924 04:03:00.375682 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:00.375686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:00.375747 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:00.375844 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:00.377512 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:00.378351 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:00.380131 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:00.380512 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:00.380636 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:00.545514 (+164878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:00.545520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:00.545523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:00.545564 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:00.547067 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:00.547994 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:00.547998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:00.548000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:00.548147 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:00.548635 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:00.548638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:00.548639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:00.548854 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:00.548933 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:00.549726 (+ 793us) server_negotiation.cc:299] Negotiation successful
0924 04:03:00.549852 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:03:00.551282 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:00.373615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46493 (local address 127.0.98.1:57859)
0924 04:03:00.374144 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:00.374200 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:00.375467 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:00.376085 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:00.376096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:00.376640 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:00.377331 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:00.377351 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:00.378509 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:00.378513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:00.379973 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:00.379986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:00.380690 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:00.380693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:00.380860 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:00.381548 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:00.381566 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:00.545202 (+163636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:00.547240 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:00.547249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:00.547272 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:00.547764 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:00.548251 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:00.548254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:00.548256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:00.548507 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:00.548970 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:00.548979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:00.549401 (+ 422us) client_negotiation.cc:769] Sending connection context
0924 04:03:00.549663 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 04:03:00.550077 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":124,"threads_started":1}
I20260924 04:03:00.552783 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46493
I20260924 04:03:00.553045 1153 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:00.553603 1153 heartbeater.cc:511] Master 127.0.98.62:46493 requested a full tablet report, sending...
I20260924 04:03:00.555168 962 ts_manager.cc:194] Registered new tserver with Master: d4e9a77aa8644abd993fc8317bc56fb2 (127.0.98.1:37915)
I20260924 04:03:00.556404 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260924 04:03:01.366075 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.992s user 0.001s sys 0.002s
I20260924 04:03:01.366600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46493
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:01.483825 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:01.484139 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:01.484184 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:01.488300 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:01.488380 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:01.488443 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:01.494264 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:01.495394 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:01.496435 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:01.506140 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:01.506330 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:01.506330 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:01.506500 1158 server_base.cc:1034] running on GCE node
I20260924 04:03:01.506950 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:01.507619 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:01.508778 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790222581508744 us; error 44 us; skew 500 ppm
Sep 24 04:03:01 dist-test-slave-zcsk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:01.511531 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:01.512467 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46661/ using document root <none> and password file <none>
I20260924 04:03:01.513015 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:01.513082 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:01.513267 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:01.515292 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "503e979db7d5456795a993348110061d"
format_stamp: "Formatted at 2026-09-24 04:03:01 on dist-test-slave-zcsk"
server_key: "b87c1d347ed122ff4c2f289d4a4fee39"
server_key_iv: "5ebbb3eb006a9b1866723b96d2cac59d"
server_key_version: "encryptionkey@0"
I20260924 04:03:01.515836 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "503e979db7d5456795a993348110061d"
format_stamp: "Formatted at 2026-09-24 04:03:01 on dist-test-slave-zcsk"
server_key: "b87c1d347ed122ff4c2f289d4a4fee39"
server_key_iv: "5ebbb3eb006a9b1866723b96d2cac59d"
server_key_version: "encryptionkey@0"
I20260924 04:03:01.519476 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:01.522217 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:01.523590 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:01.523720 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "503e979db7d5456795a993348110061d"
format_stamp: "Formatted at 2026-09-24 04:03:01 on dist-test-slave-zcsk"
server_key: "b87c1d347ed122ff4c2f289d4a4fee39"
server_key_iv: "5ebbb3eb006a9b1866723b96d2cac59d"
server_key_version: "encryptionkey@0"
I20260924 04:03:01.523830 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:01.550845 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:01.554270 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:01.554456 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:01.555068 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:01.555979 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:01.556026 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:01.556090 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:01.556115 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:01.558832 1153 heartbeater.cc:503] Master 127.0.98.62:46493 was elected leader, sending a full tablet report...
I20260924 04:03:01.564679 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42249
I20260924 04:03:01.564774 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42249 every 8 connection(s)
I20260924 04:03:01.565645 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:01.573078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 1158
I20260924 04:03:01.573199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:01.573450 392 external_mini_cluster.cc:1428] Setting key 9256371e54fb08d5660502b76065c413
Sep 24 04:03:01 dist-test-slave-zcsk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:01.580053 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:01.567495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33073 (local address 127.0.98.62:46493)
0924 04:03:01.567800 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:01.567804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:01.568687 (+ 883us) server_negotiation.cc:407] Connection header received
0924 04:03:01.570071 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:01.570075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:01.570133 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:01.570215 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:01.571420 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:01.572059 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:01.573216 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:01.573498 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:01.573597 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:01.576418 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:01.576421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:01.576423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:01.576448 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:01.577918 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:01.578429 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:01.578432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:01.578433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:01.578513 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:01.578829 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:01.578832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:01.578834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:01.579033 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:01.579119 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:01.579708 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 04:03:01.579834 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
I20260924 04:03:01.580905 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:01.568008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46493 (local address 127.0.98.2:33073)
0924 04:03:01.568515 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:01.568567 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:01.569854 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:01.570404 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:01.570412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:01.570779 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:01.571278 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:01.571291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:01.572178 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:01.572181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:01.573093 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:01.573102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:01.573648 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:01.573652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:01.573836 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:01.574481 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:01.574498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:01.576248 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:01.578039 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:01.578046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:01.578064 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:01.578330 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:01.578597 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:01.578600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:01.578602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:01.578744 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:01.579139 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:01.579146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:01.579461 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 04:03:01.579659 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 04:03:01.580030 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:03:01.582212 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46493
I20260924 04:03:01.582470 1287 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:01.582973 1287 heartbeater.cc:511] Master 127.0.98.62:46493 requested a full tablet report, sending...
I20260924 04:03:01.584146 962 ts_manager.cc:194] Registered new tserver with Master: 503e979db7d5456795a993348110061d (127.0.98.2:42249)
I20260924 04:03:01.584758 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:03:02.587047 1287 heartbeater.cc:503] Master 127.0.98.62:46493 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260924 04:03:02.737069 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.162s user 0.000s sys 0.003s
I20260924 04:03:02.737584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46493
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:02.850920 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:02.851219 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:02.851272 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:02.855736 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:02.855829 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:02.855949 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:02.861555 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:02.862686 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:02.863729 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:02.873706 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:02.873785 1293 server_base.cc:1034] running on GCE node
W20260924 04:03:02.873731 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:02.873730 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:02.874433 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:02.875061 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:02.876212 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790222582876175 us; error 47 us; skew 500 ppm
Sep 24 04:03:02 dist-test-slave-zcsk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:02.886915 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:02.887989 1293 webserver.cc:460] Webserver started at http://127.0.98.3:43309/ using document root <none> and password file <none>
I20260924 04:03:02.888693 1293 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:02.888780 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:02.889000 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:02.891254 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c"
format_stamp: "Formatted at 2026-09-24 04:03:02 on dist-test-slave-zcsk"
server_key: "263a6d0013b6d84172aaee4201f38373"
server_key_iv: "ce0cf3fee6a4a1334ceb77c36e5eb500"
server_key_version: "encryptionkey@0"
I20260924 04:03:02.891769 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c"
format_stamp: "Formatted at 2026-09-24 04:03:02 on dist-test-slave-zcsk"
server_key: "263a6d0013b6d84172aaee4201f38373"
server_key_iv: "ce0cf3fee6a4a1334ceb77c36e5eb500"
server_key_version: "encryptionkey@0"
I20260924 04:03:02.895539 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:02.898270 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:02.899746 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:02.899864 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c"
format_stamp: "Formatted at 2026-09-24 04:03:02 on dist-test-slave-zcsk"
server_key: "263a6d0013b6d84172aaee4201f38373"
server_key_iv: "ce0cf3fee6a4a1334ceb77c36e5eb500"
server_key_version: "encryptionkey@0"
I20260924 04:03:02.900004 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:02.928483 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:02.931767 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:02.931978 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:02.932575 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:02.933408 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:02.933454 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:02.933507 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:02.933537 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:02.942224 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40001
I20260924 04:03:02.942270 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40001 every 8 connection(s)
I20260924 04:03:02.943187 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:02.944123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 1293
I20260924 04:03:02.944252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:02.944581 392 external_mini_cluster.cc:1428] Setting key 0c10472a399cf26b5880c4682bd9a959
Sep 24 04:03:02 dist-test-slave-zcsk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:02.957367 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:02.945209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48103 (local address 127.0.98.62:46493)
0924 04:03:02.945551 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:02.945556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:02.946430 (+ 874us) server_negotiation.cc:407] Connection header received
0924 04:03:02.947857 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:02.947861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:02.947948 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:02.948042 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:02.949180 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:02.949726 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:02.950928 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:02.951158 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:02.951238 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:02.953743 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:02.953746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:02.953748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:02.953773 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:02.955272 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:02.955859 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:02.955862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:02.955863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:02.955978 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:02.956290 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:02.956292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:02.956293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:02.956473 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:02.956548 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:02.957120 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 04:03:02.957226 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":157,"threads_started":1}
I20260924 04:03:02.958316 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:02.945697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46493 (local address 127.0.98.3:48103)
0924 04:03:02.946242 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:02.946299 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:02.947605 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:02.948204 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:02.948211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:02.948574 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:02.949056 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:02.949069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:02.949840 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:02.949844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:02.950758 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:02.950767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:02.951283 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:02.951286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:02.951453 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:02.952041 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:02.952056 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:02.953493 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:02.955393 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:02.955398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:02.955412 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:02.955727 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:02.956067 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:02.956070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:02.956072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:02.956211 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:02.956553 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:02.956558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:02.956864 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 04:03:02.957027 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 04:03:02.957355 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":116,"threads_started":1}
I20260924 04:03:02.959405 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46493
I20260924 04:03:02.959625 1422 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:02.960137 1422 heartbeater.cc:511] Master 127.0.98.62:46493 requested a full tablet report, sending...
I20260924 04:03:02.961072 962 ts_manager.cc:194] Registered new tserver with Master: 1b0529a6f3dc4956a4c7a4b07bc5c14c (127.0.98.3:40001)
I20260924 04:03:02.961629 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48103
I20260924 04:03:02.970762 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:03:02.981904 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:02.974189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.0.98.62:46493)
0924 04:03:02.974304 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:02.974307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:02.974511 (+ 204us) server_negotiation.cc:407] Connection header received
0924 04:03:02.974601 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:02.974604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:02.974646 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:02.974720 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:02.975430 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:02.976046 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:02.976972 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:02.977200 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:02.977287 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:02.978717 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:02.978720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:02.978723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:02.978749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:02.980151 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:02.980585 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:02.980588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:02.980590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:02.980674 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:02.981009 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:02.981012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:02.981013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:02.981255 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:02.981374 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:02.981657 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 04:03:02.981771 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:03:02.987090 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:02.990543 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:03.006791 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.001634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.0.98.1:37915)
0924 04:03:03.001872 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:03.001878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:03.002688 (+ 810us) server_negotiation.cc:407] Connection header received
0924 04:03:03.002789 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:03.002794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:03.002969 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:03.003111 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:03.003337 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.004454 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.005662 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.006278 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.006374 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.006520 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:03.006569 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:03:03.006660 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260924 04:03:03.006788 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.000360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40001 (local address 127.0.0.1:43370)
0924 04:03:03.001415 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:03.001482 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:03.001609 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:03.002358 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:03.002362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:03.002380 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:03.002554 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.002561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.003970 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.003973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.005151 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.005160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.006179 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.006182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.006340 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.006376 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:03:03.006464 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:03:03.006632 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":95,"thread_start_us":172,"threads_started":1}
I20260924 04:03:03.006785 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.000404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42249 (local address 127.0.0.1:45578)
0924 04:03:03.001787 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:03.001796 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:03.001866 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:03.002306 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:03.002312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:03.002332 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:03.002579 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.002588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.003749 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.003752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.005011 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.005020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.006168 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.006171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.006322 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.006369 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 04:03:03.006467 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:03:03.006619 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":112,"thread_start_us":151,"threads_started":1}
I20260924 04:03:03.007230 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.000680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.98.3:40001)
0924 04:03:03.001974 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:03.001983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:03.002002 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:03:03.002063 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:03.002067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:03.002231 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:03.002365 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:03.002664 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.003796 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.005266 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.006044 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.006815 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.006958 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:03.007003 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 04:03:03.007087 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":138,"threads_started":1}
I20260924 04:03:03.007761 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.001231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.0.98.2:42249)
0924 04:03:03.001591 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:03.001599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:03.001876 (+ 277us) server_negotiation.cc:407] Connection header received
0924 04:03:03.001966 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:03.001970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:03.002164 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:03.002303 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:03.002714 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.003634 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.005124 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.006044 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.006156 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.006315 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:03.007056 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 04:03:03.007623 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":72,"threads_started":1}
I20260924 04:03:03.008167 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.001227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37915 (local address 127.0.0.1:53286)
0924 04:03:03.002601 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:03.002612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:03.002693 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:03.003136 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:03.003139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:03.003157 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:03.003232 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.003238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.004578 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.004581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.005559 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.005566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.006386 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.006390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.006476 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.006494 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:03:03.006542 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:03:03.008048 (+ 1506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1318,"mutex_wait_us":1428,"thread_start_us":88,"threads_started":1}
I20260924 04:03:03.010377 1221 tablet_service.cc:1467] Processing CreateTablet for tablet ed20d7b5e53e4dd3ba523504174fac69 (DEFAULT_TABLE table=test-table [id=0608f8d614b0461ca8f58322b2a4cf86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:03.010349 1356 tablet_service.cc:1467] Processing CreateTablet for tablet ed20d7b5e53e4dd3ba523504174fac69 (DEFAULT_TABLE table=test-table [id=0608f8d614b0461ca8f58322b2a4cf86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:03.010890 1087 tablet_service.cc:1467] Processing CreateTablet for tablet ed20d7b5e53e4dd3ba523504174fac69 (DEFAULT_TABLE table=test-table [id=0608f8d614b0461ca8f58322b2a4cf86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:03.011476 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed20d7b5e53e4dd3ba523504174fac69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:03.011488 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed20d7b5e53e4dd3ba523504174fac69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:03.011862 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed20d7b5e53e4dd3ba523504174fac69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:03.017948 1441 tablet_bootstrap.cc:492] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d: Bootstrap starting.
I20260924 04:03:03.019440 1443 tablet_bootstrap.cc:492] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c: Bootstrap starting.
I20260924 04:03:03.020237 1442 tablet_bootstrap.cc:492] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2: Bootstrap starting.
I20260924 04:03:03.020378 1441 tablet_bootstrap.cc:654] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:03.021092 1441 log.cc:824] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:03.021492 1443 tablet_bootstrap.cc:654] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:03.022290 1442 tablet_bootstrap.cc:654] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:03.022753 1443 log.cc:824] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:03.023126 1441 tablet_bootstrap.cc:492] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d: No bootstrap required, opened a new log
I20260924 04:03:03.023164 1442 log.cc:824] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:03.023356 1441 ts_tablet_manager.cc:1397] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:03:03.025245 1442 tablet_bootstrap.cc:492] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2: No bootstrap required, opened a new log
I20260924 04:03:03.025436 1442 ts_tablet_manager.cc:1397] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:03:03.026001 1443 tablet_bootstrap.cc:492] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c: No bootstrap required, opened a new log
I20260924 04:03:03.026186 1443 ts_tablet_manager.cc:1397] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:03:03.028120 1441 raft_consensus.cc:348] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.028456 1441 raft_consensus.cc:374] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:03.028565 1441 raft_consensus.cc:729] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503e979db7d5456795a993348110061d, State: Initialized, Role: FOLLOWER
I20260924 04:03:03.029079 1441 consensus_queue.cc:260] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.029560 1442 raft_consensus.cc:348] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.029855 1442 raft_consensus.cc:374] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:03.029935 1442 raft_consensus.cc:729] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e9a77aa8644abd993fc8317bc56fb2, State: Initialized, Role: FOLLOWER
I20260924 04:03:03.029881 1443 raft_consensus.cc:348] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.030158 1443 raft_consensus.cc:374] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:03.030177 1441 ts_tablet_manager.cc:1428] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260924 04:03:03.030242 1443 raft_consensus.cc:729] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0529a6f3dc4956a4c7a4b07bc5c14c, State: Initialized, Role: FOLLOWER
I20260924 04:03:03.030416 1442 consensus_queue.cc:260] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.030670 1443 consensus_queue.cc:260] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.031375 1442 ts_tablet_manager.cc:1428] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:03:03.031610 1443 ts_tablet_manager.cc:1428] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:03:03.032051 1422 heartbeater.cc:503] Master 127.0.98.62:46493 was elected leader, sending a full tablet report...
W20260924 04:03:03.067559 1288 tablet.cc:2369] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:03:03.124516 1154 tablet.cc:2369] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:03:03.194125 1423 tablet.cc:2369] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:03.201676 1447 raft_consensus.cc:482] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:03.201959 1447 raft_consensus.cc:504] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.203213 1447 leader_election.cc:288] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b0529a6f3dc4956a4c7a4b07bc5c14c (127.0.98.3:40001), d4e9a77aa8644abd993fc8317bc56fb2 (127.0.98.1:37915)
I20260924 04:03:03.207132 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.203648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37061 (local address 127.0.98.1:37915)
0924 04:03:03.203787 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:03.203792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:03.203928 (+ 136us) server_negotiation.cc:407] Connection header received
0924 04:03:03.204100 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:03.204104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:03.204149 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:03.204215 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:03.204508 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.205086 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.206209 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.206610 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.206703 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.206769 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:03.206938 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:03:03.207021 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:03:03.207149 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.203522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37915 (local address 127.0.98.2:37061)
0924 04:03:03.203844 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:03.203856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:03.203992 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:03.204231 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:03.204234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:03.204250 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:03.204403 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.204409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.205202 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.205205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.206076 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.206086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.206726 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.206729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.206822 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.206845 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:03:03.206911 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:03:03.206986 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260924 04:03:03.207290 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.203726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42871 (local address 127.0.98.3:40001)
0924 04:03:03.203862 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:03.203866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:03.204166 (+ 300us) server_negotiation.cc:407] Connection header received
0924 04:03:03.204259 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:03.204261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:03.204304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:03.204371 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:03.204712 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.205240 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.206286 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.206765 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.206866 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.206927 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:03.207097 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 04:03:03.207167 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:03:03.207314 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.203523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40001 (local address 127.0.98.2:42871)
0924 04:03:03.204062 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:03.204073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:03.204173 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:03.204385 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:03.204388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:03.204403 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:03.204604 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.204609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.205345 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.205348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.206165 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:03.206174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.206884 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.206888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:03.206987 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.207004 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:03:03.207068 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:03:03.207134 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":279,"thread_start_us":114,"threads_started":1}
I20260924 04:03:03.208076 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed20d7b5e53e4dd3ba523504174fac69" candidate_uuid: "503e979db7d5456795a993348110061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" is_pre_election: true
I20260924 04:03:03.208076 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed20d7b5e53e4dd3ba523504174fac69" candidate_uuid: "503e979db7d5456795a993348110061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" is_pre_election: true
I20260924 04:03:03.208415 1376 raft_consensus.cc:2454] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 503e979db7d5456795a993348110061d in term 0.
I20260924 04:03:03.208415 1107 raft_consensus.cc:2454] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 503e979db7d5456795a993348110061d in term 0.
I20260924 04:03:03.208945 1174 leader_election.cc:302] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 503e979db7d5456795a993348110061d, d4e9a77aa8644abd993fc8317bc56fb2; no voters:
I20260924 04:03:03.209193 1447 raft_consensus.cc:2781] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:03.209273 1447 raft_consensus.cc:482] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:03.209331 1447 raft_consensus.cc:3037] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:03.227370 1447 raft_consensus.cc:504] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.227830 1447 leader_election.cc:288] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [CANDIDATE]: Term 1 election: Requested vote from peers 1b0529a6f3dc4956a4c7a4b07bc5c14c (127.0.98.3:40001), d4e9a77aa8644abd993fc8317bc56fb2 (127.0.98.1:37915)
I20260924 04:03:03.228376 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed20d7b5e53e4dd3ba523504174fac69" candidate_uuid: "503e979db7d5456795a993348110061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c"
I20260924 04:03:03.228438 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed20d7b5e53e4dd3ba523504174fac69" candidate_uuid: "503e979db7d5456795a993348110061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e9a77aa8644abd993fc8317bc56fb2"
I20260924 04:03:03.228538 1376 raft_consensus.cc:3037] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:03.228595 1107 raft_consensus.cc:3037] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:03.229694 1107 raft_consensus.cc:2454] T ed20d7b5e53e4dd3ba523504174fac69 P d4e9a77aa8644abd993fc8317bc56fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 503e979db7d5456795a993348110061d in term 1.
I20260924 04:03:03.229840 1376 raft_consensus.cc:2454] T ed20d7b5e53e4dd3ba523504174fac69 P 1b0529a6f3dc4956a4c7a4b07bc5c14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 503e979db7d5456795a993348110061d in term 1.
I20260924 04:03:03.230084 1174 leader_election.cc:302] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 503e979db7d5456795a993348110061d, d4e9a77aa8644abd993fc8317bc56fb2; no voters:
I20260924 04:03:03.230317 1447 raft_consensus.cc:2781] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:03.230563 1447 raft_consensus.cc:686] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [term 1 LEADER]: Becoming Leader. State: Replica: 503e979db7d5456795a993348110061d, State: Running, Role: LEADER
I20260924 04:03:03.230953 1447 consensus_queue.cc:237] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } }
I20260924 04:03:03.234617 962 catalog_manager.cc:5495] T ed20d7b5e53e4dd3ba523504174fac69 P 503e979db7d5456795a993348110061d reported cstate change: term changed from 0 to 1, leader changed from <none> to 503e979db7d5456795a993348110061d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "503e979db7d5456795a993348110061d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0529a6f3dc4956a4c7a4b07bc5c14c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "503e979db7d5456795a993348110061d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4e9a77aa8644abd993fc8317bc56fb2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37915 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:03:03 dist-test-slave-zcsk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:03:03.302762 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.292568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.0.98.1:37915)
0924 04:03:03.292739 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:03.292743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:03.292934 (+ 191us) server_negotiation.cc:407] Connection header received
0924 04:03:03.293022 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:03.293024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:03.293068 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:03.293129 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:03.294308 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.294818 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.295731 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.296073 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.296184 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.298872 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:03.298889 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:03.298911 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:03.298944 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:03.301014 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:03.301483 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:03.301489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:03.301491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:03.301582 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:03.301865 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:03.301868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:03.301869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:03.302165 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:03.302353 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:03.302475 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 04:03:03.302598 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 04:03:03 dist-test-slave-zcsk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:03:03 dist-test-slave-zcsk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:03:03.406870 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.397705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.0.98.1:37915)
0924 04:03:03.397867 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:03.397871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:03.398207 (+ 336us) server_negotiation.cc:407] Connection header received
0924 04:03:03.398308 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:03.398312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:03.398368 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:03.398456 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:03.399248 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.399804 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.400841 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.401067 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.401159 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.403744 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:03.403748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:03.403750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:03.403779 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:03.405376 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:03.405798 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:03.405801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:03.405803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:03.405887 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:03.406114 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:03.406116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:03.406117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:03.406281 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:03.406374 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:03.406629 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 04:03:03.406737 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 04:03:03 dist-test-slave-zcsk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:03:03 dist-test-slave-zcsk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:03:03.500715 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:03.492317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.0.98.1:37915)
0924 04:03:03.492469 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:03.492475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:03.492640 (+ 165us) server_negotiation.cc:407] Connection header received
0924 04:03:03.492773 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:03.492776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:03.492822 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:03.492889 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:03.493667 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.494417 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.495315 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:03.495537 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:03.495627 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:03.497944 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:03.497947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:03.497949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:03.497977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:03.499254 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:03.499692 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:03.499695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:03.499696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:03.499774 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:03.500037 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:03.500039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:03.500040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:03.500210 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:03.500320 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:03.500457 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 04:03:03.500551 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 04:03:03.502014 1067 tablet_service.cc:2908] Rejecting scan request for tablet ed20d7b5e53e4dd3ba523504174fac69: Uninitialized: clean time has not yet been initialized
I20260924 04:03:03.503767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 1024
I20260924 04:03:03.509589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 1158
I20260924 04:03:03.514555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 1293
I20260924 04:03:03.519225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 930
2026-09-24T04:03:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4690 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:04 dist-test-slave-zcsk krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:04 dist-test-slave-zcsk krb5kdc[1466](info): set up 2 sockets
Sep 24 04:03:04 dist-test-slave-zcsk krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260924 04:03:05.654505 392 mini_kdc.cc:121] Time spent starting KDC: real 2.114s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 04:03:05.819327 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.165s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 04:03:06.061841 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.242s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:06 dist-test-slave-zcsk krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:06.138957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35477
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:06.247851 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:06.248232 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:06.248307 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:06.253235 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:06.253305 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:03:06.253322 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:06.253340 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:06.253355 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:06.253369 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:06.258780 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35477
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:06.259831 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:06.260907 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:06.269760 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:06.269853 1479 server_base.cc:1034] running on GCE node
W20260924 04:03:06.269825 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:06.269760 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:06.270574 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:06.271739 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:06.272953 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790222586272935 us; error 76 us; skew 500 ppm
Sep 24 04:03:06 dist-test-slave-zcsk krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:06.275857 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:06.276927 1479 webserver.cc:460] Webserver started at http://127.0.98.62:39113/ using document root <none> and password file <none>
I20260924 04:03:06.277511 1479 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:06.277581 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:06.277798 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:06.280007 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "350e62b851ef41d8a9893b96f010f494"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "084c89e9f39a5222495a19b20e48e6dd"
server_key_iv: "3e58656e9e23e3a4676b8b9350f10ad6"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.280627 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "350e62b851ef41d8a9893b96f010f494"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "084c89e9f39a5222495a19b20e48e6dd"
server_key_iv: "3e58656e9e23e3a4676b8b9350f10ad6"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.284641 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:03:06.287190 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:06.288585 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:06.288726 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "350e62b851ef41d8a9893b96f010f494"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "084c89e9f39a5222495a19b20e48e6dd"
server_key_iv: "3e58656e9e23e3a4676b8b9350f10ad6"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.288841 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:06.310540 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:06.313330 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:06.313498 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:06.320763 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35477
I20260924 04:03:06.320775 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35477 every 8 connection(s)
I20260924 04:03:06.321666 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:06.324368 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:06.325081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 1479
I20260924 04:03:06.325189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:06.325457 392 external_mini_cluster.cc:1428] Setting key 2266a3c3d9b07808637033982462ccf7
I20260924 04:03:06.330155 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494: Bootstrap starting.
I20260924 04:03:06.332268 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:06.332916 1557 log.cc:824] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:03:06 dist-test-slave-zcsk krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:06.334936 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494: No bootstrap required, opened a new log
I20260924 04:03:06.338692 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350e62b851ef41d8a9893b96f010f494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } }
I20260924 04:03:06.338966 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:06.339018 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350e62b851ef41d8a9893b96f010f494, State: Initialized, Role: FOLLOWER
I20260924 04:03:06.339429 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.326807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.98.62:35477)
0924 04:03:06.327213 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:06.327228 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:06.327261 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:03:06.327878 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:06.327895 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:06.328256 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:06.328548 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:06.329270 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.330241 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.331037 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.331484 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.331629 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.333986 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:06.333992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:06.334007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:06.334035 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:06.336519 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.336934 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.336941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.336946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.337086 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.337292 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.337295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.337296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.337793 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:06.337990 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:06.338184 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 04:03:06.338469 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":131,"threads_started":1}
I20260924 04:03:06.339480 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350e62b851ef41d8a9893b96f010f494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } }
I20260924 04:03:06.339622 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:06.339675 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:06.339757 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:06.340637 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350e62b851ef41d8a9893b96f010f494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } }
I20260924 04:03:06.340947 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 350e62b851ef41d8a9893b96f010f494; no voters:
I20260924 04:03:06.341228 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:06.341312 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:06.341530 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [term 1 LEADER]: Becoming Leader. State: Replica: 350e62b851ef41d8a9893b96f010f494, State: Running, Role: LEADER
I20260924 04:03:06.341878 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350e62b851ef41d8a9893b96f010f494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } }
I20260924 04:03:06.342497 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:06.343521 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "350e62b851ef41d8a9893b96f010f494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350e62b851ef41d8a9893b96f010f494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } } }
I20260924 04:03:06.343641 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:06.344012 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 350e62b851ef41d8a9893b96f010f494. Latest consensus state: current_term: 1 leader_uuid: "350e62b851ef41d8a9893b96f010f494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350e62b851ef41d8a9893b96f010f494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35477 } } }
I20260924 04:03:06.344055 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:06.344625 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:06.346920 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:06.352344 1571 catalog_manager.cc:1347] Generated new cluster ID: 351682afca9343efb13ab15ccb17895f
I20260924 04:03:06.352407 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:06.376405 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:06.377731 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:06.390377 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494: Generated new TSK 0
I20260924 04:03:06.391294 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:06.430792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:06.547406 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:06.547674 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:06.547717 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:06.552170 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:06.552233 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:06.552274 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:06.552327 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:06.557653 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:06.558739 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:06.559770 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:06.569236 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:06.569303 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:06.569273 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:06.569411 1583 server_base.cc:1034] running on GCE node
I20260924 04:03:06.569887 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:06.570545 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:06.571731 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790222586571710 us; error 32 us; skew 500 ppm
Sep 24 04:03:06 dist-test-slave-zcsk krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:06.574594 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:06.575523 1583 webserver.cc:460] Webserver started at http://127.0.98.1:45367/ using document root <none> and password file <none>
I20260924 04:03:06.576148 1583 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:06.576221 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:06.576452 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:06.578747 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "0e6d2e2ac749e4bd4a94785d202411b4"
server_key_iv: "47976deaf3b96f35d62b32b7e64ddb5f"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.579360 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "0e6d2e2ac749e4bd4a94785d202411b4"
server_key_iv: "47976deaf3b96f35d62b32b7e64ddb5f"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.583174 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:03:06.585796 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:06.587190 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:06.587304 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "0e6d2e2ac749e4bd4a94785d202411b4"
server_key_iv: "47976deaf3b96f35d62b32b7e64ddb5f"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.587414 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:06.609975 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:06.612790 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:06.612963 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:06.614403 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:06.614454 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:06.614507 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:06.614533 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:03:06 dist-test-slave-zcsk krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:06.624651 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42413
I20260924 04:03:06.624680 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42413 every 8 connection(s)
I20260924 04:03:06.625737 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:06.627328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 1583
I20260924 04:03:06.627367 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.615113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51887 (local address 127.0.98.62:35477)
0924 04:03:06.615229 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:06.615233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:06.616305 (+ 1072us) server_negotiation.cc:407] Connection header received
0924 04:03:06.617279 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:06.617284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:06.617342 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:06.617399 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:06.618828 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.619383 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.620602 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.620847 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.620944 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.623673 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:06.623677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:06.623680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:06.623710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:06.625103 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.625626 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.625629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.625630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.625701 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.626079 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.626082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.626083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.626334 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:06.626491 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:06.627055 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 04:03:06.627207 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:03:06.627439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:06.627688 392 external_mini_cluster.cc:1428] Setting key 24470400ed63ce9760be52770a0e3b9e
I20260924 04:03:06.628204 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.615674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.1:51887)
0924 04:03:06.616166 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:06.616207 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:06.617043 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:06.617817 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:06.617826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:06.618168 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:06.618689 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:06.618709 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.619504 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.619507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:06.620488 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:06.620498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.620996 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.621000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:06.621169 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.621728 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:06.621750 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:06.623477 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:06.625206 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:06.625212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:06.625226 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:06.625527 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:06.625776 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:06.625780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:06.625782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:06.625956 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:06.626543 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:06.626549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:06.626784 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 04:03:06.626988 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 04:03:06.627243 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":159,"threads_started":1}
I20260924 04:03:06.629837 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35477
I20260924 04:03:06.629987 1714 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:06.630546 1714 heartbeater.cc:511] Master 127.0.98.62:35477 requested a full tablet report, sending...
I20260924 04:03:06.632218 1510 ts_manager.cc:194] Registered new tserver with Master: f113e853df4f46eaa6a2a8e4aa11f0e5 (127.0.98.1:42413)
I20260924 04:03:06.633358 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51887
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:03:06.640290 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.632084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37569 (local address 127.0.98.62:35477)
0924 04:03:06.632197 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:06.632201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:06.634104 (+ 1903us) server_negotiation.cc:407] Connection header received
0924 04:03:06.634207 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:06.634210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:06.634256 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:06.634332 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:06.634831 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.635328 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.636164 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.636381 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.636464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.637488 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:06.637492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:06.637494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:06.637520 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:06.638736 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.639143 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.639146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.639148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.639218 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.639450 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.639453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.639455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.639622 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:06.639702 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:06.640028 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 04:03:06.640135 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:03:06.640331 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.632033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.1:37569)
0924 04:03:06.634015 (+ 1982us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:06.634029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:06.634129 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:06.634364 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:06.634366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:06.634530 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:06.634681 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:06.634690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.635447 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.635449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:06.636062 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:06.636071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.636477 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.636481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:06.636574 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.637028 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:06.637039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:06.637364 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:06.638837 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:06.638840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:06.638843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:06.639059 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:06.639296 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:06.639298 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:06.639301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:06.639375 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:06.639708 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:06.639711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:06.639827 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 04:03:06.639973 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 04:03:06.640119 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1894,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:03:06.655149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:06.761994 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:06.762287 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:06.762332 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:06.766465 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:06.766525 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:06.766566 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:06.766604 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:06.771795 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:06.772797 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:06.773751 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:06.783264 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:06.783264 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:06.783316 1723 server_base.cc:1034] running on GCE node
W20260924 04:03:06.783330 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:06.783937 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:06.784610 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:06.785764 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790222586785745 us; error 33 us; skew 500 ppm
Sep 24 04:03:06 dist-test-slave-zcsk krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:06.788439 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:06.789300 1723 webserver.cc:460] Webserver started at http://127.0.98.2:41425/ using document root <none> and password file <none>
I20260924 04:03:06.789835 1723 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:06.789901 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:06.790086 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:06.792188 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "71f4e55455ce4b38af065cca300df9cd"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "13bdb9e62cb2692b9086dff1a690e843"
server_key_iv: "5ec10b6412ce5c30719f936e0a6658af"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.792773 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "71f4e55455ce4b38af065cca300df9cd"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "13bdb9e62cb2692b9086dff1a690e843"
server_key_iv: "5ec10b6412ce5c30719f936e0a6658af"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.796415 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:03:06.798985 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:06.800236 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:06.800354 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "71f4e55455ce4b38af065cca300df9cd"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "13bdb9e62cb2692b9086dff1a690e843"
server_key_iv: "5ec10b6412ce5c30719f936e0a6658af"
server_key_version: "encryptionkey@0"
I20260924 04:03:06.800448 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:06.819177 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:06.822362 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:06.822554 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:06.824049 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:06.824105 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:06.824164 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:06.824191 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:03:06 dist-test-slave-zcsk krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:06.833887 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35147
I20260924 04:03:06.834228 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35147 every 8 connection(s)
I20260924 04:03:06.834906 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:06.837230 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.824931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52941 (local address 127.0.98.62:35477)
0924 04:03:06.825076 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:06.825086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:06.826013 (+ 927us) server_negotiation.cc:407] Connection header received
0924 04:03:06.827139 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:06.827143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:06.827198 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:06.827271 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:06.828557 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.829080 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.830335 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.830554 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.830642 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.833679 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:06.833683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:06.833686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:06.833714 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:06.835167 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.835750 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.835754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.835756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.835846 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.836176 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.836179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.836181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.836366 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:06.836470 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:06.836941 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 04:03:06.837072 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:03:06.837975 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.825378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.2:52941)
0924 04:03:06.825844 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:06.825894 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:06.826911 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:06.827513 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:06.827523 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:06.827884 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:06.828415 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:06.828429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.829205 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.829209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:06.830217 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:06.830233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.830788 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.830792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:06.830936 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.832052 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:06.832069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:06.833513 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:06.835270 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:06.835276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:06.835291 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:06.835621 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:06.835950 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:06.835953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:06.835955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:06.836089 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:06.836465 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:06.836471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:06.836716 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 04:03:06.836871 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:03:06.837179 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260924 04:03:06.839176 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35477
I20260924 04:03:06.839305 1854 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:06.839773 1854 heartbeater.cc:511] Master 127.0.98.62:35477 requested a full tablet report, sending...
I20260924 04:03:06.840785 1511 ts_manager.cc:194] Registered new tserver with Master: 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147)
I20260924 04:03:06.840961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 1723
I20260924 04:03:06.841048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:06.841301 392 external_mini_cluster.cc:1428] Setting key 399793cc06984301baacf5db8cbac269
I20260924 04:03:06.841528 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52941
I20260924 04:03:06.849385 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.841493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56441 (local address 127.0.98.62:35477)
0924 04:03:06.841595 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:06.841598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:06.842431 (+ 833us) server_negotiation.cc:407] Connection header received
0924 04:03:06.842471 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:06.842474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:06.842516 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:06.842597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:06.843389 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.844037 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.845245 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.845518 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.845617 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.846639 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:06.846643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:06.846645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:06.846666 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:06.847891 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.848284 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.848288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.848290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.848362 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:06.848603 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:06.848607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:06.848608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:06.848809 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:06.848895 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:06.849135 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 04:03:06.849233 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:03:06.849395 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:06.841691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.2:56441)
0924 04:03:06.842223 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:06.842239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:06.842344 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:06.842621 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:06.842624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:06.842842 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:06.842973 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:06.842981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.844187 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.844191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:06.845082 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:06.845094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:06.845630 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:06.845633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:06.845703 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:06.846185 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:06.846197 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:06.846508 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:06.848005 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:06.848008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:06.848011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:06.848201 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:06.848440 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:06.848443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:06.848446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:06.848523 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:06.848897 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:06.848900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:06.848988 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 04:03:06.849073 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:03:06.849211 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:03:06.867166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35477
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:06.973939 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:06.974208 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:06.974253 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:06.978308 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:06.978387 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:06.978432 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:06.978466 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:06.983773 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:06.984860 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:06.985834 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:06.995743 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:06.995852 1863 server_base.cc:1034] running on GCE node
W20260924 04:03:06.995828 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:06.995759 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:06.996495 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:06.997164 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:06.998346 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790222586998309 us; error 45 us; skew 500 ppm
Sep 24 04:03:06 dist-test-slave-zcsk krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:07.001252 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:07.002274 1863 webserver.cc:460] Webserver started at http://127.0.98.3:43631/ using document root <none> and password file <none>
I20260924 04:03:07.002869 1863 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:07.002938 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:07.003161 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:07.005275 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "c7e95bad01424ce48bbb5bb1818a228b"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "b4cab2ef89bab1aafda07fc142c2a8bb"
server_key_iv: "972c7eaff230ebb59c603953233f1d3b"
server_key_version: "encryptionkey@0"
I20260924 04:03:07.005805 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c7e95bad01424ce48bbb5bb1818a228b"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "b4cab2ef89bab1aafda07fc142c2a8bb"
server_key_iv: "972c7eaff230ebb59c603953233f1d3b"
server_key_version: "encryptionkey@0"
I20260924 04:03:07.009608 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:03:07.012077 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:07.013468 1863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:07.013579 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "c7e95bad01424ce48bbb5bb1818a228b"
format_stamp: "Formatted at 2026-09-24 04:03:06 on dist-test-slave-zcsk"
server_key: "b4cab2ef89bab1aafda07fc142c2a8bb"
server_key_iv: "972c7eaff230ebb59c603953233f1d3b"
server_key_version: "encryptionkey@0"
I20260924 04:03:07.013690 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:07.032655 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:07.035882 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:07.036095 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:07.037542 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:07.037590 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:07.037631 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:07.037657 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:03:07 dist-test-slave-zcsk krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:07.047842 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34173
I20260924 04:03:07.047880 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34173 every 8 connection(s)
I20260924 04:03:07.048851 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:07.051492 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.038475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58071 (local address 127.0.98.62:35477)
0924 04:03:07.038614 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.038620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.039491 (+ 871us) server_negotiation.cc:407] Connection header received
0924 04:03:07.040812 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.040816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.040873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.040949 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:07.042498 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.043349 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.044827 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.045204 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.045332 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.048188 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:07.048191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:07.048194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:07.048221 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:07.049732 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.050207 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.050210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.050212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.050293 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.050537 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.050539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.050540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.050711 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:07.050781 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.051257 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.051355 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:03:07.052285 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.038883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.3:58071)
0924 04:03:07.039340 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.039391 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.040581 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.041167 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.041176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.041604 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:07.042321 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.042343 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.043524 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.043529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.044706 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.044718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.045376 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.045379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.045577 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.046231 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:07.046251 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:07.047998 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:07.049834 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:07.049838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:07.049848 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:07.050101 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:07.050350 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:07.050352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:07.050353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:07.050470 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:07.050782 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:07.050786 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:07.051037 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.051182 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.051433 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260924 04:03:07.052918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 1863
I20260924 04:03:07.053025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:07.053285 392 external_mini_cluster.cc:1428] Setting key 9ee098c5a3909b80d78a55eb68e88291
I20260924 04:03:07.053529 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35477
I20260924 04:03:07.053682 1994 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:07.054253 1994 heartbeater.cc:511] Master 127.0.98.62:35477 requested a full tablet report, sending...
I20260924 04:03:07.055253 1510 ts_manager.cc:194] Registered new tserver with Master: c7e95bad01424ce48bbb5bb1818a228b (127.0.98.3:34173)
I20260924 04:03:07.055976 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58071
I20260924 04:03:07.062106 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.055785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48477 (local address 127.0.98.62:35477)
0924 04:03:07.055885 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.055889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.056131 (+ 242us) server_negotiation.cc:407] Connection header received
0924 04:03:07.056254 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.056257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.056299 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.056361 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:07.056763 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.057257 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.058029 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.058249 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.058329 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.059338 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:07.059342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:07.059344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:07.059368 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:07.060674 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.061047 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.061050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.061052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.061127 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.061355 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.061358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.061360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.061528 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:07.061609 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.061842 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.061941 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:03:07.062122 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.055715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.98.3:48477)
0924 04:03:07.056060 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.056073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.056170 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.056386 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.056388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.056575 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:07.056665 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.056670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.057366 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.057370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.057921 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.057937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.058339 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.058342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.058427 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.058882 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:07.058896 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:07.059218 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:07.060771 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:07.060774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:07.060776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:07.060965 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:07.061202 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:07.061204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:07.061207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:07.061282 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:07.061610 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:07.061613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:07.061699 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.061781 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.061915 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":92,"threads_started":1}
I20260924 04:03:07.066941 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:03:07 dist-test-slave-zcsk krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:03:07 dist-test-slave-zcsk krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:07.093104 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.084456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.98.62:35477)
0924 04:03:07.084610 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.084614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.084628 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:07.084669 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.084671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.084711 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.084781 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:07.085542 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.086105 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.086955 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.087170 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.087262 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.089819 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:07.089823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:07.089825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:07.089855 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:07.091273 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.091730 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.091734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.091735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.091843 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.092186 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.092189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.092191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.092400 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:07.092530 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.092827 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.092950 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:03:07.096381 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:07.099323 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:07.113605 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.107800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.0.98.2:35147)
0924 04:03:07.108581 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.108593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.108615 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:03:07.108690 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.108695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.108896 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.109063 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:07.109399 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.110400 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.112068 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.112901 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.113010 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.113185 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.113360 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.113487 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":475,"threads_started":1}
I20260924 04:03:07.113622 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.107644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35147 (local address 127.0.0.1:37348)
0924 04:03:07.108073 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.108118 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.108235 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.109050 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.109056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.109074 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:07.109185 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.109190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.110505 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.110509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.111966 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.111976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.113035 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.113039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.113180 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.113225 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.113330 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.113468 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":31,"thread_start_us":63,"threads_started":1}
I20260924 04:03:07.114121 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.108102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42413 (local address 127.0.0.1:34240)
0924 04:03:07.109267 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.109280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.109363 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.109908 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.109911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.109929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:07.110016 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.110021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.111073 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.111077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.112055 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.112064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.113035 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.113039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.113819 (+ 780us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.113842 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.113922 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.113991 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260924 04:03:07.114583 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.108450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.0.98.1:42413)
0924 04:03:07.108690 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.108704 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.109449 (+ 745us) server_negotiation.cc:407] Connection header received
0924 04:03:07.109519 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.109523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.109665 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.109777 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:07.110134 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.110974 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.112166 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.112907 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.113010 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.113176 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.114294 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.114397 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260924 04:03:07.115290 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.108409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.0.98.3:34173)
0924 04:03:07.108791 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.108800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.108990 (+ 190us) server_negotiation.cc:407] Connection header received
0924 04:03:07.109396 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.109402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.109584 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.109729 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:07.109942 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.110793 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.113265 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.114036 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.114133 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.114405 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.114586 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.114941 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":118,"threads_started":1}
I20260924 04:03:07.115413 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.107758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34173 (local address 127.0.0.1:50966)
0924 04:03:07.108894 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.108906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.109277 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.109728 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.109731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.109749 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:07.109840 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.109846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.110916 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.110919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.113138 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.113146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.114143 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.114147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.114399 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.114421 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.114491 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.114555 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":1058,"thread_start_us":109,"threads_started":1}
I20260924 04:03:07.116854 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdb998786c346069495e842de6171bb (DEFAULT_TABLE table=test-table [id=102bc0c6e20b4ff5aedc0e61b9e968bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:07.118021 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdb998786c346069495e842de6171bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.118024 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdb998786c346069495e842de6171bb (DEFAULT_TABLE table=test-table [id=102bc0c6e20b4ff5aedc0e61b9e968bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:07.119048 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdb998786c346069495e842de6171bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.118994 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 3cdb998786c346069495e842de6171bb (DEFAULT_TABLE table=test-table [id=102bc0c6e20b4ff5aedc0e61b9e968bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:07.119777 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cdb998786c346069495e842de6171bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.125991 2017 tablet_bootstrap.cc:492] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Bootstrap starting.
I20260924 04:03:07.126984 2018 tablet_bootstrap.cc:492] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Bootstrap starting.
I20260924 04:03:07.127470 2019 tablet_bootstrap.cc:492] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: Bootstrap starting.
I20260924 04:03:07.128391 2017 tablet_bootstrap.cc:654] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.128746 2018 tablet_bootstrap.cc:654] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.129222 2017 log.cc:824] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:07.129418 2019 tablet_bootstrap.cc:654] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.129410 2018 log.cc:824] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:07.130165 2019 log.cc:824] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:07.131331 2017 tablet_bootstrap.cc:492] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: No bootstrap required, opened a new log
I20260924 04:03:07.131331 2018 tablet_bootstrap.cc:492] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: No bootstrap required, opened a new log
I20260924 04:03:07.131531 2018 ts_tablet_manager.cc:1397] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:03:07.131531 2017 ts_tablet_manager.cc:1397] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:03:07.131945 2019 tablet_bootstrap.cc:492] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: No bootstrap required, opened a new log
I20260924 04:03:07.132117 2019 ts_tablet_manager.cc:1397] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:03:07.134779 2019 raft_consensus.cc:348] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
I20260924 04:03:07.135010 2019 raft_consensus.cc:374] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.135080 2019 raft_consensus.cc:729] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f113e853df4f46eaa6a2a8e4aa11f0e5, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.135381 2017 raft_consensus.cc:348] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
I20260924 04:03:07.135509 2019 consensus_queue.cc:260] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
I20260924 04:03:07.135661 2017 raft_consensus.cc:374] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.135741 2017 raft_consensus.cc:729] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f4e55455ce4b38af065cca300df9cd, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.135370 2018 raft_consensus.cc:348] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
I20260924 04:03:07.135893 2018 raft_consensus.cc:374] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.136047 2018 raft_consensus.cc:729] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e95bad01424ce48bbb5bb1818a228b, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.136269 1714 heartbeater.cc:503] Master 127.0.98.62:35477 was elected leader, sending a full tablet report...
I20260924 04:03:07.136248 2017 consensus_queue.cc:260] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
I20260924 04:03:07.136590 2019 ts_tablet_manager.cc:1428] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 04:03:07.136555 2018 consensus_queue.cc:260] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
I20260924 04:03:07.137205 1854 heartbeater.cc:503] Master 127.0.98.62:35477 was elected leader, sending a full tablet report...
I20260924 04:03:07.137416 1994 heartbeater.cc:503] Master 127.0.98.62:35477 was elected leader, sending a full tablet report...
I20260924 04:03:07.138525 2017 ts_tablet_manager.cc:1428] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:03:07.138859 2018 ts_tablet_manager.cc:1428] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260924 04:03:07.285619 2023 raft_consensus.cc:482] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:07.285848 2023 raft_consensus.cc:504] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
I20260924 04:03:07.286996 2023 leader_election.cc:288] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147), c7e95bad01424ce48bbb5bb1818a228b (127.0.98.3:34173)
I20260924 04:03:07.291050 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.287395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51579 (local address 127.0.98.3:34173)
0924 04:03:07.287534 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.287538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.287960 (+ 422us) server_negotiation.cc:407] Connection header received
0924 04:03:07.288098 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.288102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.288148 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.288218 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:07.288526 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.289030 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.290079 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.290467 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.290556 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.290636 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.290853 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.290923 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:03:07.291061 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.287403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52949 (local address 127.0.98.2:35147)
0924 04:03:07.287546 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.287550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.287661 (+ 111us) server_negotiation.cc:407] Connection header received
0924 04:03:07.287803 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.287806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.287850 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.287945 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:07.288321 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.288911 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.290054 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.290463 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.290563 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.290625 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.290867 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.290939 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:03:07.291078 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.287282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34173 (local address 127.0.98.1:51579)
0924 04:03:07.287834 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.287845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.287983 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.288288 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.288292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.288310 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:07.288418 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.288425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.289136 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.289138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.289945 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.289954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.290615 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.290619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.290730 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.290755 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.290824 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.290903 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":319,"thread_start_us":51,"threads_started":1}
I20260924 04:03:07.291085 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.287282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35147 (local address 127.0.98.1:52949)
0924 04:03:07.287553 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.287568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.287681 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.288031 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.288034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.288054 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:07.288190 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.288198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.289021 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.289024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.289922 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.289939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.290600 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.290604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.290703 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.290753 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.290817 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.290891 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":57,"thread_start_us":95,"threads_started":1}
I20260924 04:03:07.291705 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdb998786c346069495e842de6171bb" candidate_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e95bad01424ce48bbb5bb1818a228b" is_pre_election: true
I20260924 04:03:07.291706 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdb998786c346069495e842de6171bb" candidate_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f4e55455ce4b38af065cca300df9cd" is_pre_election: true
I20260924 04:03:07.292025 1808 raft_consensus.cc:2454] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f113e853df4f46eaa6a2a8e4aa11f0e5 in term 0.
I20260924 04:03:07.292028 1948 raft_consensus.cc:2454] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f113e853df4f46eaa6a2a8e4aa11f0e5 in term 0.
I20260924 04:03:07.292451 1600 leader_election.cc:302] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7e95bad01424ce48bbb5bb1818a228b, f113e853df4f46eaa6a2a8e4aa11f0e5; no voters:
I20260924 04:03:07.292706 2023 raft_consensus.cc:2781] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:07.292785 2023 raft_consensus.cc:482] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:07.292817 2023 raft_consensus.cc:3037] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.293846 2023 raft_consensus.cc:504] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
I20260924 04:03:07.294198 2023 leader_election.cc:288] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147), c7e95bad01424ce48bbb5bb1818a228b (127.0.98.3:34173)
I20260924 04:03:07.294540 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdb998786c346069495e842de6171bb" candidate_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f4e55455ce4b38af065cca300df9cd"
I20260924 04:03:07.294598 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cdb998786c346069495e842de6171bb" candidate_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e95bad01424ce48bbb5bb1818a228b"
I20260924 04:03:07.294647 1808 raft_consensus.cc:3037] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.294739 1948 raft_consensus.cc:3037] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.295758 1808 raft_consensus.cc:2454] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f113e853df4f46eaa6a2a8e4aa11f0e5 in term 1.
I20260924 04:03:07.296008 1948 raft_consensus.cc:2454] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f113e853df4f46eaa6a2a8e4aa11f0e5 in term 1.
I20260924 04:03:07.296058 1599 leader_election.cc:302] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f4e55455ce4b38af065cca300df9cd, f113e853df4f46eaa6a2a8e4aa11f0e5; no voters:
I20260924 04:03:07.296216 2023 raft_consensus.cc:2781] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:07.296437 2023 raft_consensus.cc:686] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 LEADER]: Becoming Leader. State: Replica: f113e853df4f46eaa6a2a8e4aa11f0e5, State: Running, Role: LEADER
I20260924 04:03:07.296756 2023 consensus_queue.cc:237] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } }
W20260924 04:03:07.299819 1995 tablet.cc:2369] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:07.299990 1510 catalog_manager.cc:5495] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f113e853df4f46eaa6a2a8e4aa11f0e5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } health_report { overall_health: HEALTHY } } }
W20260924 04:03:07.336246 1855 tablet.cc:2369] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:07.354631 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.350937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.0.98.1:42413)
0924 04:03:07.351078 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.351083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.351097 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:07.351260 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.351263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.351317 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.351396 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:07.351683 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.352251 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.353146 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.353359 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.353444 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.353775 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:07.354236 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:07.354330 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.354431 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.354490 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:03:07.360438 1808 raft_consensus.cc:1261] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Refusing update from remote peer f113e853df4f46eaa6a2a8e4aa11f0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:07.360476 1948 raft_consensus.cc:1261] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Refusing update from remote peer f113e853df4f46eaa6a2a8e4aa11f0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:07.361095 2023 consensus_queue.cc:1035] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:07.361279 2028 consensus_queue.cc:1035] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:07.370647 2033 mvcc.cc:205] Tried to move back new op lower bound from 7332751717821427712 to 7332751717570224128. Current Snapshot: MvccSnapshot[applied={T|T < 7332751717821427712}]
I20260924 04:03:07.380021 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:07.380218 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:07.382797 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494: Sending DeleteTablet for 3 replicas of tablet 3cdb998786c346069495e842de6171bb
I20260924 04:03:07.383598 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdb998786c346069495e842de6171bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37348
I20260924 04:03:07.383613 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdb998786c346069495e842de6171bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50966
I20260924 04:03:07.384405 2041 tablet_replica.cc:330] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: stopping tablet replica
I20260924 04:03:07.384373 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:03:07.384464 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cdb998786c346069495e842de6171bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260924 04:03:07.384730 2041 raft_consensus.cc:2229] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 04:03:07.384781 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:07.385030 2043 tablet_replica.cc:330] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: stopping tablet replica
I20260924 04:03:07.385067 2041 raft_consensus.cc:2258] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:07.385347 2043 raft_consensus.cc:2229] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:07.385740 2043 raft_consensus.cc:2258] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:07.386591 2041 ts_tablet_manager.cc:1905] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:07.387089 2043 ts_tablet_manager.cc:1905] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:07.389297 2041 ts_tablet_manager.cc:1918] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:07.389393 2041 log.cc:1196] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/3cdb998786c346069495e842de6171bb
I20260924 04:03:07.389671 2041 ts_tablet_manager.cc:1939] T 3cdb998786c346069495e842de6171bb P 71f4e55455ce4b38af065cca300df9cd: Deleting consensus metadata
I20260924 04:03:07.389786 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 35fdb21624724185a80131743201becb (DEFAULT_TABLE table=test-table [id=b26009249d3341c89fff2cb5c74969b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:07.390026 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fdb21624724185a80131743201becb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.389967 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 35fdb21624724185a80131743201becb (DEFAULT_TABLE table=test-table [id=b26009249d3341c89fff2cb5c74969b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:07.390188 2042 tablet_replica.cc:330] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: stopping tablet replica
I20260924 04:03:07.390219 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fdb21624724185a80131743201becb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.390482 2042 raft_consensus.cc:2229] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:07.390782 2042 raft_consensus.cc:2258] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:07.392513 2018 tablet_bootstrap.cc:492] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: Bootstrap starting.
I20260924 04:03:07.392678 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 35fdb21624724185a80131743201becb (DEFAULT_TABLE table=test-table [id=b26009249d3341c89fff2cb5c74969b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:07.392899 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fdb21624724185a80131743201becb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.393509 2018 tablet_bootstrap.cc:654] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.393844 1495 catalog_manager.cc:4841] TS 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147): tablet 3cdb998786c346069495e842de6171bb (table test-table [id=102bc0c6e20b4ff5aedc0e61b9e968bb]) successfully deleted
I20260924 04:03:07.394291 2019 tablet_bootstrap.cc:492] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: Bootstrap starting.
I20260924 04:03:07.395210 2019 tablet_bootstrap.cc:654] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.395812 2017 tablet_bootstrap.cc:492] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: Bootstrap starting.
I20260924 04:03:07.395841 2042 ts_tablet_manager.cc:1905] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:07.395998 2043 ts_tablet_manager.cc:1918] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:07.396107 2043 log.cc:1196] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/3cdb998786c346069495e842de6171bb
I20260924 04:03:07.396406 2043 ts_tablet_manager.cc:1939] T 3cdb998786c346069495e842de6171bb P f113e853df4f46eaa6a2a8e4aa11f0e5: Deleting consensus metadata
I20260924 04:03:07.396404 2019 tablet_bootstrap.cc:492] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: No bootstrap required, opened a new log
I20260924 04:03:07.396565 2019 ts_tablet_manager.cc:1397] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:07.396868 2017 tablet_bootstrap.cc:654] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.396987 2019 raft_consensus.cc:348] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.397094 2019 raft_consensus.cc:374] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.397125 2019 raft_consensus.cc:729] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f113e853df4f46eaa6a2a8e4aa11f0e5, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.397240 2019 consensus_queue.cc:260] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.397505 2019 ts_tablet_manager.cc:1428] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:07.398175 2042 ts_tablet_manager.cc:1918] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:07.398377 1498 catalog_manager.cc:4841] TS f113e853df4f46eaa6a2a8e4aa11f0e5 (127.0.98.1:42413): tablet 3cdb998786c346069495e842de6171bb (table test-table [id=102bc0c6e20b4ff5aedc0e61b9e968bb]) successfully deleted
I20260924 04:03:07.398926 2042 log.cc:1196] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/3cdb998786c346069495e842de6171bb
I20260924 04:03:07.399230 2017 tablet_bootstrap.cc:492] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: No bootstrap required, opened a new log
I20260924 04:03:07.399282 2042 ts_tablet_manager.cc:1939] T 3cdb998786c346069495e842de6171bb P c7e95bad01424ce48bbb5bb1818a228b: Deleting consensus metadata
I20260924 04:03:07.399322 2017 ts_tablet_manager.cc:1397] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:03:07.399751 2017 raft_consensus.cc:348] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.399857 2017 raft_consensus.cc:374] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.398548 2018 tablet_bootstrap.cc:492] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: No bootstrap required, opened a new log
I20260924 04:03:07.400151 2018 ts_tablet_manager.cc:1397] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260924 04:03:07.400431 1496 catalog_manager.cc:4841] TS c7e95bad01424ce48bbb5bb1818a228b (127.0.98.3:34173): tablet 3cdb998786c346069495e842de6171bb (table test-table [id=102bc0c6e20b4ff5aedc0e61b9e968bb]) successfully deleted
I20260924 04:03:07.400583 2018 raft_consensus.cc:348] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.399886 2017 raft_consensus.cc:729] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f4e55455ce4b38af065cca300df9cd, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.400681 2018 raft_consensus.cc:374] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.400712 2018 raft_consensus.cc:729] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e95bad01424ce48bbb5bb1818a228b, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.400779 2017 consensus_queue.cc:260] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.400813 2018 consensus_queue.cc:260] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.401026 2018 ts_tablet_manager.cc:1428] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:07.401039 2017 ts_tablet_manager.cc:1428] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:07.454403 2025 raft_consensus.cc:482] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:07.454550 2025 raft_consensus.cc:504] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.455755 2025 leader_election.cc:288] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f113e853df4f46eaa6a2a8e4aa11f0e5 (127.0.98.1:42413), 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147)
I20260924 04:03:07.460116 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.456450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36549 (local address 127.0.98.1:42413)
0924 04:03:07.456569 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.456573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.456586 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:07.456633 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.456636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.456680 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.456749 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:07.457001 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.457502 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.459176 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.459588 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.459688 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.459754 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.459921 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.459989 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:03:07.460126 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.455958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42413 (local address 127.0.98.3:36549)
0924 04:03:07.456215 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.456230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.456311 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.456755 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.456758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.456772 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:07.456894 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.456901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.458207 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.458210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.459046 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.459054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.459691 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.459695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.459791 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.459815 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.459878 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.459988 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":18}
I20260924 04:03:07.460763 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.456093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35147 (local address 127.0.98.3:42399)
0924 04:03:07.456489 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.456501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.456588 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.456874 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.456877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.456894 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:07.457489 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.457498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.458197 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.458200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.459441 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.459451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.460368 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.460372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.460465 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.460488 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.460563 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.460625 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260924 04:03:07.460741 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb21624724185a80131743201becb" candidate_uuid: "c7e95bad01424ce48bbb5bb1818a228b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" is_pre_election: true
I20260924 04:03:07.460965 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.456403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42399 (local address 127.0.98.2:35147)
0924 04:03:07.456524 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.456528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.456546 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:03:07.456715 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.456718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.456766 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.456827 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:07.457621 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.458098 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.459567 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.460154 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.460247 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.460308 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.460776 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.460856 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:03:07.460969 1668 raft_consensus.cc:2454] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e95bad01424ce48bbb5bb1818a228b in term 0.
I20260924 04:03:07.461364 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb21624724185a80131743201becb" candidate_uuid: "c7e95bad01424ce48bbb5bb1818a228b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f4e55455ce4b38af065cca300df9cd" is_pre_election: true
I20260924 04:03:07.461359 1882 leader_election.cc:302] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7e95bad01424ce48bbb5bb1818a228b, f113e853df4f46eaa6a2a8e4aa11f0e5; no voters:
I20260924 04:03:07.461494 1808 raft_consensus.cc:2454] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e95bad01424ce48bbb5bb1818a228b in term 0.
I20260924 04:03:07.461601 2025 raft_consensus.cc:2781] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:07.461650 2025 raft_consensus.cc:482] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:07.461683 2025 raft_consensus.cc:3037] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.462659 2025 raft_consensus.cc:504] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.462935 2025 leader_election.cc:288] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [CANDIDATE]: Term 1 election: Requested vote from peers f113e853df4f46eaa6a2a8e4aa11f0e5 (127.0.98.1:42413), 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147)
I20260924 04:03:07.463253 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb21624724185a80131743201becb" candidate_uuid: "c7e95bad01424ce48bbb5bb1818a228b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5"
I20260924 04:03:07.463263 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb21624724185a80131743201becb" candidate_uuid: "c7e95bad01424ce48bbb5bb1818a228b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f4e55455ce4b38af065cca300df9cd"
I20260924 04:03:07.463361 1668 raft_consensus.cc:3037] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.463382 1808 raft_consensus.cc:3037] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.464315 1808 raft_consensus.cc:2454] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e95bad01424ce48bbb5bb1818a228b in term 1.
I20260924 04:03:07.464490 1668 raft_consensus.cc:2454] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e95bad01424ce48bbb5bb1818a228b in term 1.
I20260924 04:03:07.464583 1879 leader_election.cc:302] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f4e55455ce4b38af065cca300df9cd, c7e95bad01424ce48bbb5bb1818a228b; no voters:
I20260924 04:03:07.464747 2025 raft_consensus.cc:2781] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:07.464943 2025 raft_consensus.cc:686] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 1 LEADER]: Becoming Leader. State: Replica: c7e95bad01424ce48bbb5bb1818a228b, State: Running, Role: LEADER
I20260924 04:03:07.465180 2025 consensus_queue.cc:237] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.467337 1510 catalog_manager.cc:5495] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b reported cstate change: term changed from 0 to 1, leader changed from <none> to c7e95bad01424ce48bbb5bb1818a228b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } health_report { overall_health: HEALTHY } } }
Sep 24 04:03:07 dist-test-slave-zcsk krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:07.498631 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.489588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.0.98.62:35477)
0924 04:03:07.489718 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.489723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.489918 (+ 195us) server_negotiation.cc:407] Connection header received
0924 04:03:07.490073 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.490075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.490114 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.490180 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:07.491049 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.491618 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.492785 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.493013 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.493090 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.495561 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:07.495564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:07.495566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:07.495586 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:07.496819 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.497335 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.497338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.497339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.497405 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.497666 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.497668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.497669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.497819 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:07.497884 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.498223 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.498443 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":50}
I20260924 04:03:07.498646 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.489508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35477 (local address 127.0.0.1:45906)
0924 04:03:07.489824 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.489841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.489981 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.490230 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.490233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.490797 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:07.490919 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.490927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.491742 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.491746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.492680 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.492690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.493129 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.493133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.493228 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.493834 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:07.493848 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:07.495436 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:07.496923 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:07.496932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:07.496939 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:07.497245 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:07.497485 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:07.497488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:07.497490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:07.497588 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:07.497919 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:07.497926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:07.498064 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.498191 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.498393 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260924 04:03:07.502502 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45906:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:03:07.503343 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:07.508190 1788 tablet_service.cc:1467] Processing CreateTablet for tablet a88cbad979434371888a3c21fd0c7570 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5d7af296cba4a46bef7826ea12a2db8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:07.508225 1648 tablet_service.cc:1467] Processing CreateTablet for tablet a88cbad979434371888a3c21fd0c7570 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5d7af296cba4a46bef7826ea12a2db8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:07.508225 1928 tablet_service.cc:1467] Processing CreateTablet for tablet a88cbad979434371888a3c21fd0c7570 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5d7af296cba4a46bef7826ea12a2db8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:07.508591 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88cbad979434371888a3c21fd0c7570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.508591 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88cbad979434371888a3c21fd0c7570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.508595 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88cbad979434371888a3c21fd0c7570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:07.511229 2019 tablet_bootstrap.cc:492] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: Bootstrap starting.
I20260924 04:03:07.511335 2018 tablet_bootstrap.cc:492] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: Bootstrap starting.
I20260924 04:03:07.512313 2018 tablet_bootstrap.cc:654] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.512776 2019 tablet_bootstrap.cc:654] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.513574 2017 tablet_bootstrap.cc:492] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: Bootstrap starting.
I20260924 04:03:07.514166 2019 tablet_bootstrap.cc:492] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: No bootstrap required, opened a new log
I20260924 04:03:07.514254 2019 ts_tablet_manager.cc:1397] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:03:07.514340 2017 tablet_bootstrap.cc:654] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:07.514714 2019 raft_consensus.cc:348] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.514818 2019 raft_consensus.cc:374] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.514846 2019 raft_consensus.cc:729] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f113e853df4f46eaa6a2a8e4aa11f0e5, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.514994 2019 consensus_queue.cc:260] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.515218 2036 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.515332 2036 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:07.515604 2019 ts_tablet_manager.cc:1428] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:03:07.516436 2017 tablet_bootstrap.cc:492] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: No bootstrap required, opened a new log
I20260924 04:03:07.516512 2017 ts_tablet_manager.cc:1397] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:07.516716 2018 tablet_bootstrap.cc:492] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: No bootstrap required, opened a new log
I20260924 04:03:07.516811 2018 ts_tablet_manager.cc:1397] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 04:03:07.516878 2017 raft_consensus.cc:348] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.516979 2017 raft_consensus.cc:374] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.517005 2017 raft_consensus.cc:729] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f4e55455ce4b38af065cca300df9cd, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.517107 2017 consensus_queue.cc:260] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.517313 2017 ts_tablet_manager.cc:1428] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:07.517251 2018 raft_consensus.cc:348] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.517367 2018 raft_consensus.cc:374] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:07.517401 2018 raft_consensus.cc:729] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e95bad01424ce48bbb5bb1818a228b, State: Initialized, Role: FOLLOWER
I20260924 04:03:07.517516 2018 consensus_queue.cc:260] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.517517 2024 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.517627 2024 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:07.517784 2018 ts_tablet_manager.cc:1428] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:07.517989 2025 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.518097 2025 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 04:03:07.627745 1715 tablet.cc:2369] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:07.798810 2025 raft_consensus.cc:482] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:07.798939 2025 raft_consensus.cc:504] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.799329 2025 leader_election.cc:288] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f113e853df4f46eaa6a2a8e4aa11f0e5 (127.0.98.1:42413), 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147)
I20260924 04:03:07.799786 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88cbad979434371888a3c21fd0c7570" candidate_uuid: "c7e95bad01424ce48bbb5bb1818a228b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f4e55455ce4b38af065cca300df9cd" is_pre_election: true
I20260924 04:03:07.799786 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88cbad979434371888a3c21fd0c7570" candidate_uuid: "c7e95bad01424ce48bbb5bb1818a228b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" is_pre_election: true
I20260924 04:03:07.799960 1808 raft_consensus.cc:2454] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e95bad01424ce48bbb5bb1818a228b in term 0.
I20260924 04:03:07.799961 1668 raft_consensus.cc:2454] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e95bad01424ce48bbb5bb1818a228b in term 0.
I20260924 04:03:07.800236 1879 leader_election.cc:302] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f4e55455ce4b38af065cca300df9cd, c7e95bad01424ce48bbb5bb1818a228b; no voters:
I20260924 04:03:07.800413 2025 raft_consensus.cc:2781] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:07.800472 2025 raft_consensus.cc:482] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:07.800498 2025 raft_consensus.cc:3037] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.801680 2025 raft_consensus.cc:504] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.801971 2025 leader_election.cc:288] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [CANDIDATE]: Term 1 election: Requested vote from peers f113e853df4f46eaa6a2a8e4aa11f0e5 (127.0.98.1:42413), 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147)
I20260924 04:03:07.802287 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88cbad979434371888a3c21fd0c7570" candidate_uuid: "c7e95bad01424ce48bbb5bb1818a228b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f4e55455ce4b38af065cca300df9cd"
I20260924 04:03:07.802294 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88cbad979434371888a3c21fd0c7570" candidate_uuid: "c7e95bad01424ce48bbb5bb1818a228b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5"
I20260924 04:03:07.802386 1808 raft_consensus.cc:3037] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.802428 1668 raft_consensus.cc:3037] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:07.803186 1808 raft_consensus.cc:2454] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e95bad01424ce48bbb5bb1818a228b in term 1.
I20260924 04:03:07.803419 1668 raft_consensus.cc:2454] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e95bad01424ce48bbb5bb1818a228b in term 1.
I20260924 04:03:07.803428 1879 leader_election.cc:302] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f4e55455ce4b38af065cca300df9cd, c7e95bad01424ce48bbb5bb1818a228b; no voters:
I20260924 04:03:07.803586 2025 raft_consensus.cc:2781] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:07.803668 2025 raft_consensus.cc:686] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [term 1 LEADER]: Becoming Leader. State: Replica: c7e95bad01424ce48bbb5bb1818a228b, State: Running, Role: LEADER
I20260924 04:03:07.803798 2025 consensus_queue.cc:237] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } }
I20260924 04:03:07.804479 2048 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: TxnStatusTablet state changed. Reason: New leader c7e95bad01424ce48bbb5bb1818a228b. Latest consensus state: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.804586 2048 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:07.804792 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:03:07.804993 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:03:07.805538 1510 catalog_manager.cc:5495] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b reported cstate change: term changed from 0 to 1, leader changed from <none> to c7e95bad01424ce48bbb5bb1818a228b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } health_report { overall_health: HEALTHY } } }
I20260924 04:03:07.846891 2048 consensus_queue.cc:1035] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:07.851431 2048 consensus_queue.cc:1035] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:07.881158 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.877111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.0.98.3:34173)
0924 04:03:07.877296 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.877299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.877332 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:03:07.877531 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.877533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.877600 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.877692 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:07.877939 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.878569 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.879777 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.880036 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.880114 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.880401 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:07.880819 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:07.880901 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.880988 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.881036 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":54,"threads_started":1}
I20260924 04:03:07.881193 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.877033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34173 (local address 127.0.0.1:35832)
0924 04:03:07.877273 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.877289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.877403 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.877714 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.877717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.877726 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:07.877832 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.877838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.878669 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.878672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.879674 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.879684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.880127 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.880131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.880221 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.880281 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:07.880896 (+ 615us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:07.880908 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.880968 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.881053 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 04:03:07.883982 1808 raft_consensus.cc:1261] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Refusing update from remote peer c7e95bad01424ce48bbb5bb1818a228b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:07.884084 1668 raft_consensus.cc:1261] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 FOLLOWER]: Refusing update from remote peer c7e95bad01424ce48bbb5bb1818a228b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:07.884466 2025 consensus_queue.cc:1035] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:07.884622 2048 consensus_queue.cc:1035] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:07.885872 2036 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: TxnStatusTablet state changed. Reason: New leader c7e95bad01424ce48bbb5bb1818a228b. Latest consensus state: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.885995 2036 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:07.886200 2024 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: TxnStatusTablet state changed. Reason: New leader c7e95bad01424ce48bbb5bb1818a228b. Latest consensus state: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.886305 2024 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:07.887427 2048 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.887528 2048 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:07.887622 2032 mvcc.cc:205] Tried to move back new op lower bound from 7332751719968907264 to 7332751719645908992. Current Snapshot: MvccSnapshot[applied={T|T < 7332751719968907264}]
I20260924 04:03:07.888923 2024 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.889117 2024 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P 71f4e55455ce4b38af065cca300df9cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:07.889046 2025 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.889554 2025 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P c7e95bad01424ce48bbb5bb1818a228b: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:07.893811 2036 tablet_replica.cc:439] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7e95bad01424ce48bbb5bb1818a228b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f113e853df4f46eaa6a2a8e4aa11f0e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42413 } } peers { permanent_uuid: "71f4e55455ce4b38af065cca300df9cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35147 } } peers { permanent_uuid: "c7e95bad01424ce48bbb5bb1818a228b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34173 } } }
I20260924 04:03:07.893957 2036 tablet_replica.cc:442] T a88cbad979434371888a3c21fd0c7570 P f113e853df4f46eaa6a2a8e4aa11f0e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:07.894119 2031 mvcc.cc:205] Tried to move back new op lower bound from 7332751719968907264 to 7332751719645908992. Current Snapshot: MvccSnapshot[applied={T|T < 7332751719968907264}]
I20260924 04:03:07.896988 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.893261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.0.98.3:34173)
0924 04:03:07.893482 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.893486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.893499 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:07.893538 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.893540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.893640 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.893710 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:07.893952 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.894687 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.895653 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.895887 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.896005 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.896351 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:07.896431 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:07.896489 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.896818 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.896884 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260924 04:03:07.908025 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.903765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53777 (local address 127.0.98.3:34173)
0924 04:03:07.903861 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.903864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.904577 (+ 713us) server_negotiation.cc:407] Connection header received
0924 04:03:07.904745 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.904749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.904799 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.904878 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:07.905123 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.905912 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.906773 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.907129 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.907228 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.907529 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:07.907609 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:07.907677 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.907807 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.907854 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":36}
I20260924 04:03:07.908152 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.903724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34173 (local address 127.0.98.3:53777)
0924 04:03:07.904499 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:07.904514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:07.904608 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:07.904886 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:07.904888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:07.904894 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:07.905006 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.905010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.906032 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.906035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.906671 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:07.906679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.907227 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.907229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:07.907337 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.907417 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:07.907709 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:07.907715 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:03:07.907781 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:03:07.907848 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":21,"thread_start_us":546,"threads_started":1}
I20260924 04:03:07.949710 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:07.949867 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:07.951875 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 350e62b851ef41d8a9893b96f010f494: Sending DeleteTablet for 3 replicas of tablet 35fdb21624724185a80131743201becb
I20260924 04:03:07.952523 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fdb21624724185a80131743201becb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50966
I20260924 04:03:07.952543 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fdb21624724185a80131743201becb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37348
I20260924 04:03:07.952656 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fdb21624724185a80131743201becb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260924 04:03:07.952929 2080 tablet_replica.cc:330] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: stopping tablet replica
I20260924 04:03:07.953078 2080 raft_consensus.cc:2229] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:07.953194 2080 raft_consensus.cc:2258] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:07.953449 2078 tablet_replica.cc:330] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: stopping tablet replica
I20260924 04:03:07.953573 2078 raft_consensus.cc:2229] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:07.953691 2078 raft_consensus.cc:2258] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:07.953910 2079 tablet_replica.cc:330] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: stopping tablet replica
I20260924 04:03:07.954073 2079 raft_consensus.cc:2229] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:07.954269 2080 ts_tablet_manager.cc:1905] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:07.954487 2079 raft_consensus.cc:2258] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:07.955124 2079 ts_tablet_manager.cc:1905] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:07.957384 2080 ts_tablet_manager.cc:1918] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:07.957477 2080 log.cc:1196] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/35fdb21624724185a80131743201becb
I20260924 04:03:07.957733 2080 ts_tablet_manager.cc:1939] T 35fdb21624724185a80131743201becb P f113e853df4f46eaa6a2a8e4aa11f0e5: Deleting consensus metadata
I20260924 04:03:07.957782 2079 ts_tablet_manager.cc:1918] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:07.957888 2079 log.cc:1196] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/35fdb21624724185a80131743201becb
I20260924 04:03:07.958098 2079 ts_tablet_manager.cc:1939] T 35fdb21624724185a80131743201becb P c7e95bad01424ce48bbb5bb1818a228b: Deleting consensus metadata
I20260924 04:03:07.958474 1498 catalog_manager.cc:4841] TS f113e853df4f46eaa6a2a8e4aa11f0e5 (127.0.98.1:42413): tablet 35fdb21624724185a80131743201becb (table test-table [id=b26009249d3341c89fff2cb5c74969b5]) successfully deleted
I20260924 04:03:07.958693 1496 catalog_manager.cc:4841] TS c7e95bad01424ce48bbb5bb1818a228b (127.0.98.3:34173): tablet 35fdb21624724185a80131743201becb (table test-table [id=b26009249d3341c89fff2cb5c74969b5]) successfully deleted
I20260924 04:03:07.959079 2078 ts_tablet_manager.cc:1905] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:07.961709 2078 ts_tablet_manager.cc:1918] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:07.961784 2078 log.cc:1196] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/35fdb21624724185a80131743201becb
I20260924 04:03:07.962020 2078 ts_tablet_manager.cc:1939] T 35fdb21624724185a80131743201becb P 71f4e55455ce4b38af065cca300df9cd: Deleting consensus metadata
I20260924 04:03:07.962625 1495 catalog_manager.cc:4841] TS 71f4e55455ce4b38af065cca300df9cd (127.0.98.2:35147): tablet 35fdb21624724185a80131743201becb (table test-table [id=b26009249d3341c89fff2cb5c74969b5]) successfully deleted
Sep 24 04:03:07 dist-test-slave-zcsk krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:03:07.975024 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.954362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.0.98.1:42413)
0924 04:03:07.954469 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.954473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.958962 (+ 4489us) server_negotiation.cc:407] Connection header received
0924 04:03:07.959075 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.959078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.959127 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.959210 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:07.960070 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.960880 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.962132 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.962412 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.962505 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.971159 (+ 8654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:07.971174 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:07.971194 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:07.971230 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:07.973118 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.973677 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.973683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.973685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.973774 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.974037 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.974040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.974042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.974305 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:07.974538 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.974733 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.974851 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:03:07.975500 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48730
I20260924 04:03:07.984964 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:07.977162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.0.98.62:35477)
0924 04:03:07.977282 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:07.977285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:07.977366 (+ 81us) server_negotiation.cc:407] Connection header received
0924 04:03:07.977491 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:07.977493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:07.977532 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:07.977596 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:07.978405 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.978944 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.980087 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:07.980357 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:07.980461 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:07.982036 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:07.982040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:07.982044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:07.982073 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:07.983446 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.983832 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.983836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.983838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.983943 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:07.984173 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:07.984176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:07.984178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:07.984365 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:07.984476 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:07.984696 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 04:03:07.984806 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 04:03:07.985373 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58012
I20260924 04:03:07.986588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 1583
I20260924 04:03:07.996706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 1723
I20260924 04:03:08.002211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 1863
I20260924 04:03:08.007556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 1479
2026-09-24T04:03:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4488 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:08 dist-test-slave-zcsk krb5kdc[2087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:08 dist-test-slave-zcsk krb5kdc[2087](info): set up 2 sockets
Sep 24 04:03:08 dist-test-slave-zcsk krb5kdc[2087](info): commencing operation
krb5kdc: starting...
W20260924 04:03:08.959404 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:09.077028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38615
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:09.185484 2100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:09.185774 2100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:09.185817 2100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:09.189934 2100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:09.190007 2100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:03:09.190024 2100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:09.190040 2100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:09.190055 2100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:09.190068 2100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:09.195868 2100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38615
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:09.197064 2100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:09.198117 2100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:09.207022 2106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:09.207067 2100 server_base.cc:1034] running on GCE node
W20260924 04:03:09.207085 2105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:09.207031 2108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:09.207780 2100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:09.208961 2100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:09.210124 2100 hybrid_clock.cc:648] HybridClock initialized: now 1790222589210105 us; error 29 us; skew 500 ppm
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:09.212810 2100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:09.213778 2100 webserver.cc:460] Webserver started at http://127.0.98.62:33587/ using document root <none> and password file <none>
I20260924 04:03:09.214296 2100 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:09.214352 2100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:09.214519 2100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:09.216570 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "694912fc2e4f4ffaa9c8fbe75230595f"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "4838102a6a97b3ae90049973fc02ea8a"
server_key_iv: "91d1b35fb89fda65b179765c9c1bb38f"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.217078 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "694912fc2e4f4ffaa9c8fbe75230595f"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "4838102a6a97b3ae90049973fc02ea8a"
server_key_iv: "91d1b35fb89fda65b179765c9c1bb38f"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.220606 2100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:03:09.223042 2115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.224390 2100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:03:09.224516 2100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "694912fc2e4f4ffaa9c8fbe75230595f"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "4838102a6a97b3ae90049973fc02ea8a"
server_key_iv: "91d1b35fb89fda65b179765c9c1bb38f"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.224617 2100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:09.248229 2100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:09.251494 2100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:09.251686 2100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:09.259416 2100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38615
I20260924 04:03:09.259464 2177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38615 every 8 connection(s)
I20260924 04:03:09.260630 2100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:09.263051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 2100
I20260924 04:03:09.263163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:09.263442 392 external_mini_cluster.cc:1428] Setting key 62123a0040bd9984ba2eb359d628c0a0
I20260924 04:03:09.263756 2178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:09.270035 2178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: Bootstrap starting.
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:09.272300 2178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:09.273159 2178 log.cc:824] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:09.275089 2178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: No bootstrap required, opened a new log
I20260924 04:03:09.277802 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.264823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.0.98.62:38615)
0924 04:03:09.265232 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:09.265252 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:09.265454 (+ 202us) server_negotiation.cc:407] Connection header received
0924 04:03:09.266331 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:09.266352 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:09.266711 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:09.267050 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:09.267822 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.268864 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.269710 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.270212 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.270358 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.272838 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:09.272843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:09.272857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:09.272884 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:09.275018 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.275479 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.275484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.275489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.275610 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.275878 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.275880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.275882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.276333 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:09.276517 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:09.276782 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 04:03:09.277057 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260924 04:03:09.278247 2178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694912fc2e4f4ffaa9c8fbe75230595f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38615 } }
I20260924 04:03:09.278486 2178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:09.278550 2178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694912fc2e4f4ffaa9c8fbe75230595f, State: Initialized, Role: FOLLOWER
I20260924 04:03:09.279019 2178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694912fc2e4f4ffaa9c8fbe75230595f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38615 } }
I20260924 04:03:09.279184 2178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:09.279238 2178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:09.279330 2178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:09.280483 2178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694912fc2e4f4ffaa9c8fbe75230595f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38615 } }
I20260924 04:03:09.280791 2178 leader_election.cc:302] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694912fc2e4f4ffaa9c8fbe75230595f; no voters:
I20260924 04:03:09.281045 2178 leader_election.cc:288] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:09.281183 2183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:09.281392 2183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [term 1 LEADER]: Becoming Leader. State: Replica: 694912fc2e4f4ffaa9c8fbe75230595f, State: Running, Role: LEADER
I20260924 04:03:09.281675 2183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694912fc2e4f4ffaa9c8fbe75230595f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38615 } }
I20260924 04:03:09.282188 2178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:09.283195 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "694912fc2e4f4ffaa9c8fbe75230595f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694912fc2e4f4ffaa9c8fbe75230595f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38615 } } }
I20260924 04:03:09.283255 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 694912fc2e4f4ffaa9c8fbe75230595f. Latest consensus state: current_term: 1 leader_uuid: "694912fc2e4f4ffaa9c8fbe75230595f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694912fc2e4f4ffaa9c8fbe75230595f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38615 } } }
I20260924 04:03:09.283344 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:09.283344 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:09.284157 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:09.287245 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:09.292373 2192 catalog_manager.cc:1347] Generated new cluster ID: bb0c2779f22e400eb53d287daa49a5a8
I20260924 04:03:09.292449 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:09.302281 2192 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:09.303167 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:09.314018 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: Generated new TSK 0
I20260924 04:03:09.314852 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:09.347003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38615
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:09.455246 2204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:09.455492 2204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:09.455541 2204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:09.459476 2204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:09.459539 2204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:09.459581 2204 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:03:09.459610 2204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:09.459658 2204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:09.464942 2204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:09.465937 2204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:09.466926 2204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:09.476500 2204 server_base.cc:1034] running on GCE node
W20260924 04:03:09.476511 2209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:09.476656 2212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:09.476511 2210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:09.477097 2204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:09.477712 2204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:09.478858 2204 hybrid_clock.cc:648] HybridClock initialized: now 1790222589478831 us; error 41 us; skew 500 ppm
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:09.481480 2204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:09.482378 2204 webserver.cc:460] Webserver started at http://127.0.98.1:45785/ using document root <none> and password file <none>
I20260924 04:03:09.482923 2204 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:09.482987 2204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:09.483202 2204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:09.485277 2204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "b186b40397ec41faaa08dd73495aecbb"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "e735736e00bd88dc9b63240b3fdb733d"
server_key_iv: "878d60dee58aac163b2199af853be63d"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.485785 2204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b186b40397ec41faaa08dd73495aecbb"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "e735736e00bd88dc9b63240b3fdb733d"
server_key_iv: "878d60dee58aac163b2199af853be63d"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.489289 2204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:09.491562 2219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.492862 2204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:09.493019 2204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "b186b40397ec41faaa08dd73495aecbb"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "e735736e00bd88dc9b63240b3fdb733d"
server_key_iv: "878d60dee58aac163b2199af853be63d"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.493132 2204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:09.515563 2204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:09.518610 2204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:09.518774 2204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:09.520190 2204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:09.520238 2204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.520287 2204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:09.520301 2204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:09.530488 2204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33265
I20260924 04:03:09.530570 2334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33265 every 8 connection(s)
I20260924 04:03:09.531577 2204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:09.532640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 2204
I20260924 04:03:09.532738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:09.533017 392 external_mini_cluster.cc:1428] Setting key cd1f59442a97a2f6b1490e2115f15917
I20260924 04:03:09.533406 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.520914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57611 (local address 127.0.98.62:38615)
0924 04:03:09.521059 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:09.521064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:09.521974 (+ 910us) server_negotiation.cc:407] Connection header received
0924 04:03:09.523171 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:09.523175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:09.523238 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:09.523364 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:09.524675 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.525245 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.526436 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.526698 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.526794 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.529523 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:09.529527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:09.529529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:09.529555 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:09.531202 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.531682 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.531686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.531687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.531787 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.532166 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.532169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.532171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.532445 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:09.532541 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:09.533114 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 04:03:09.533242 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:03:09.534312 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.521305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38615 (local address 127.0.98.1:57611)
0924 04:03:09.521782 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:09.521851 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:09.522919 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:09.523559 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:09.523567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:09.523975 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:09.524527 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.524544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.525376 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.525379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.526315 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.526325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.526860 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.526864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.527020 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.527581 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:09.527602 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:09.529359 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:09.531307 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.531312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.531323 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.531560 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.531872 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.531875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.531876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.532071 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.532587 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:09.532593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:09.532881 (+ 288us) client_negotiation.cc:769] Sending connection context
0924 04:03:09.533038 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:03:09.533365 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
I20260924 04:03:09.535604 2335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38615
I20260924 04:03:09.535763 2335 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:09.536327 2335 heartbeater.cc:511] Master 127.0.98.62:38615 requested a full tablet report, sending...
I20260924 04:03:09.537827 2131 ts_manager.cc:194] Registered new tserver with Master: b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265)
I20260924 04:03:09.539037 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57611
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:03:09.544818 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.537528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38615 (local address 127.0.98.1:41511)
0924 04:03:09.537793 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:09.537808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:09.537890 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:09.538094 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:09.538097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:09.538281 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:09.538397 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.538403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.539393 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.539397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.540208 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.540218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.540827 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.540830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.540919 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.541510 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:09.541521 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:09.541843 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:09.543542 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.543545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.543547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.543741 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.544007 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.544009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.544010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.544079 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.544381 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:09.544411 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:09.544483 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 04:03:09.544556 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:03:09.544656 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":6,"thread_start_us":80,"threads_started":1}
I20260924 04:03:09.544836 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.537790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41511 (local address 127.0.98.62:38615)
0924 04:03:09.537897 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:09.537900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:09.537914 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:09.537953 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:09.537956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:09.537995 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:09.538068 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:09.538501 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.539263 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.540340 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.540699 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.540813 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.541978 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:09.541982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:09.541985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:09.542010 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:09.543439 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.543842 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.543845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.543846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.543933 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.544144 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.544146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.544147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.544307 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:09.544378 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:09.544609 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:03:09.544700 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:03:09.563300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38615
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:09.671657 2344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:09.671964 2344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:09.672014 2344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:09.675980 2344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:09.676048 2344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:09.676087 2344 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:03:09.676110 2344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:09.676146 2344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:09.681510 2344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:09.682655 2344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:09.683763 2344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:09.693513 2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:09.693516 2349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:09.693516 2350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:09.693773 2344 server_base.cc:1034] running on GCE node
I20260924 04:03:09.694208 2344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:09.694872 2344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:09.696010 2344 hybrid_clock.cc:648] HybridClock initialized: now 1790222589695978 us; error 43 us; skew 500 ppm
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:09.698861 2344 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:09.699788 2344 webserver.cc:460] Webserver started at http://127.0.98.2:41863/ using document root <none> and password file <none>
I20260924 04:03:09.700367 2344 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:09.700438 2344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:09.700639 2344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:09.702699 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "26264b2caebf4120abcceb6211ef6530"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "ce75f19822f841e3f258f58e3ecbe9b2"
server_key_iv: "ebda785dd4285f8432cb18867ede48e4"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.703248 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "26264b2caebf4120abcceb6211ef6530"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "ce75f19822f841e3f258f58e3ecbe9b2"
server_key_iv: "ebda785dd4285f8432cb18867ede48e4"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.706974 2344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:03:09.709548 2359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.710812 2344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:09.710925 2344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "26264b2caebf4120abcceb6211ef6530"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "ce75f19822f841e3f258f58e3ecbe9b2"
server_key_iv: "ebda785dd4285f8432cb18867ede48e4"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.711032 2344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:09.721109 2344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:09.724200 2344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:09.724371 2344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:09.725759 2344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:09.725809 2344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.725863 2344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:09.725891 2344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.735383 2344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35825
I20260924 04:03:09.735424 2474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35825 every 8 connection(s)
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:09.736498 2344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:09.739149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 2344
I20260924 04:03:09.739281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:09.739580 392 external_mini_cluster.cc:1428] Setting key e45fdbb208d26bc9d872dfa414e1c398
I20260924 04:03:09.740226 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.726503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43661 (local address 127.0.98.62:38615)
0924 04:03:09.726678 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:09.726684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:09.727517 (+ 833us) server_negotiation.cc:407] Connection header received
0924 04:03:09.728688 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:09.728692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:09.728773 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:09.728873 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:09.730658 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.731469 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.733187 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.733490 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.733594 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.736508 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:09.736511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:09.736513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:09.736539 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:09.738070 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.738653 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.738657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.738658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.738742 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.739115 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.739118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.739120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.739366 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:09.739466 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:09.739973 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 04:03:09.740076 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:03:09.741256 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.726914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38615 (local address 127.0.98.2:43661)
0924 04:03:09.727361 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:09.727408 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:09.728422 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:09.729269 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:09.729278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:09.729770 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:09.730494 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.730515 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.731617 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.731621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.733064 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.733075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.733663 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.733668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.733894 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.734530 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:09.734550 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:09.736337 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:09.738253 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.738258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.738270 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.738514 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.738846 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.738849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.738850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.738990 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.739459 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:09.739464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:09.739704 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 04:03:09.739867 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 04:03:09.740210 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260924 04:03:09.742552 2475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38615
I20260924 04:03:09.742750 2475 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:09.743238 2475 heartbeater.cc:511] Master 127.0.98.62:38615 requested a full tablet report, sending...
I20260924 04:03:09.744227 2132 ts_manager.cc:194] Registered new tserver with Master: 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825)
I20260924 04:03:09.744786 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43661
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:03:09.754966 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.748347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55497 (local address 127.0.98.62:38615)
0924 04:03:09.748477 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:09.748481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:09.748621 (+ 140us) server_negotiation.cc:407] Connection header received
0924 04:03:09.748708 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:09.748710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:09.748748 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:09.748811 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:09.749243 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.749738 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.750603 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.750850 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.750948 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.752013 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:09.752017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:09.752020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:09.752045 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:09.753367 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.753826 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.753829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.753831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.753920 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.754193 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.754195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.754197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.754410 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:09.754484 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:09.754751 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 04:03:09.754844 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:03:09.755156 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.748249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38615 (local address 127.0.98.2:55497)
0924 04:03:09.748551 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:09.748563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:09.748649 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:09.748832 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:09.748834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:09.748993 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:09.749136 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.749143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.749857 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.749859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.750492 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.750503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.750953 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.750955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.751026 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.751510 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:09.751521 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:09.751857 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:09.753479 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.753483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.753485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.753724 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.754014 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.754018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.754020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.754116 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.754496 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:09.754499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:09.754601 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:03:09.754715 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:03:09.754862 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:03:09.766652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38615
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:09.879590 2484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:09.879856 2484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:09.879925 2484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:09.884021 2484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:09.884091 2484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:09.884150 2484 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:03:09.884179 2484 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:09.884241 2484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:09.889658 2484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:09.890687 2484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:09.891706 2484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:09.900779 2492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:09.900787 2489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:09.900898 2490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:09.900945 2484 server_base.cc:1034] running on GCE node
I20260924 04:03:09.901578 2484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:09.902163 2484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:09.903323 2484 hybrid_clock.cc:648] HybridClock initialized: now 1790222589903305 us; error 32 us; skew 500 ppm
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:09.905844 2484 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:09.906617 2484 webserver.cc:460] Webserver started at http://127.0.98.3:44515/ using document root <none> and password file <none>
I20260924 04:03:09.907124 2484 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:09.907181 2484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:09.907351 2484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:09.909343 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "34688313822c433db985899599b188d7"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "74ec20e5b3537afeed860b8cccce1581"
server_key_iv: "4c5bfa471b55f72cd9deddde5e7d7cd3"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.909821 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "34688313822c433db985899599b188d7"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "74ec20e5b3537afeed860b8cccce1581"
server_key_iv: "4c5bfa471b55f72cd9deddde5e7d7cd3"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.913380 2484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:09.915688 2499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.916942 2484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:09.917039 2484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "34688313822c433db985899599b188d7"
format_stamp: "Formatted at 2026-09-24 04:03:09 on dist-test-slave-zcsk"
server_key: "74ec20e5b3537afeed860b8cccce1581"
server_key_iv: "4c5bfa471b55f72cd9deddde5e7d7cd3"
server_key_version: "encryptionkey@0"
I20260924 04:03:09.917143 2484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:09.942955 2484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:09.945940 2484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:09.946089 2484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:09.947427 2484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:09.947481 2484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.947535 2484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:09.947562 2484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:09.956988 2484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37231
I20260924 04:03:09.957067 2614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37231 every 8 connection(s)
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:09.957988 2484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:09.961081 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.948237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34957 (local address 127.0.98.62:38615)
0924 04:03:09.948340 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:09.948343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:09.949441 (+ 1098us) server_negotiation.cc:407] Connection header received
0924 04:03:09.950492 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:09.950496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:09.950557 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:09.950651 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:09.952182 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.953010 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.954721 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.954956 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.955060 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.957983 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:09.957986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:09.957988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:09.958015 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:09.959421 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.959848 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.959850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.959852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.959957 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.960221 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.960223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.960224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.960389 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:09.960456 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:09.960868 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 04:03:09.960963 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:03:09.961910 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.948829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38615 (local address 127.0.98.3:34957)
0924 04:03:09.949301 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:09.949343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:09.950250 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:09.950940 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:09.950949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:09.951362 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:09.952037 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.952058 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.953148 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.953152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.954585 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.954595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.955127 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.955131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.955359 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.956096 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:09.956115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:09.957814 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:09.959514 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.959518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.959529 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.959771 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.960028 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.960030 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.960031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.960154 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.960457 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:09.960462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:09.960679 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 04:03:09.960838 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:03:09.961081 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260924 04:03:09.962909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 2484
I20260924 04:03:09.963003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:09.963063 2615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38615
I20260924 04:03:09.963194 2615 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:09.963342 392 external_mini_cluster.cc:1428] Setting key 5ec60acf997950d4c7ac21a6e6e43fab
I20260924 04:03:09.963732 2615 heartbeater.cc:511] Master 127.0.98.62:38615 requested a full tablet report, sending...
I20260924 04:03:09.964830 2131 ts_manager.cc:194] Registered new tserver with Master: 34688313822c433db985899599b188d7 (127.0.98.3:37231)
I20260924 04:03:09.965484 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34957
I20260924 04:03:09.971067 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.965072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53265 (local address 127.0.98.62:38615)
0924 04:03:09.965179 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:09.965183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:09.965405 (+ 222us) server_negotiation.cc:407] Connection header received
0924 04:03:09.965508 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:09.965511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:09.965552 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:09.965616 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:09.966044 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.966534 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.967304 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.967508 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.967587 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.968564 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:09.968568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:09.968570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:09.968593 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:09.969672 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.970043 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.970046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.970048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.970119 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:09.970352 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:09.970355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:09.970356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:09.970517 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:09.970596 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:09.970831 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 04:03:09.970931 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:03:09.971077 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.965091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38615 (local address 127.0.98.3:53265)
0924 04:03:09.965339 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:09.965353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:09.965440 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:09.965629 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:09.965631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:09.965835 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:09.965953 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.965961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.966645 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.966649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.967204 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:09.967212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.967597 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.967600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:09.967685 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.968121 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:09.968131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:09.968447 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:09.969778 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.969780 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.969783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.969961 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.970193 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:09.970203 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:09.970205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:09.970277 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:09.970598 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:09.970601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:09.970686 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 04:03:09.970771 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:03:09.970900 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260924 04:03:09.976939 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:03:09 dist-test-slave-zcsk krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:10.002063 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:09.993795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.0.98.62:38615)
0924 04:03:09.993942 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:09.993946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:09.993960 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:09.994002 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:09.994005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:09.994045 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:09.994121 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:09.994875 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.995440 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.996372 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:09.996617 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:09.996711 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:09.999178 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:09.999181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:09.999184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:09.999213 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:10.000667 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:10.001083 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:10.001086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:10.001088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:10.001174 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:10.001381 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:10.001383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:10.001385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:10.001559 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:10.001653 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.001837 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.001939 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:03:10.004992 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:10.008113 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:10.025322 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.019372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33265 (local address 127.0.0.1:34322)
0924 04:03:10.019613 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.019626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.019706 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.020684 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.020691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.020709 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:10.020860 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.020867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.022360 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.022363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.023778 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.023789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.024823 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.024827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.024930 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.024968 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.025051 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.025168 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
I20260924 04:03:10.025338 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.020220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37231 (local address 127.0.0.1:51886)
0924 04:03:10.020483 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.020496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.020699 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.021474 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.021477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.021493 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:10.021630 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.021637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.022960 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.022963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.023728 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.023736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.025020 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.025023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.025112 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.025132 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.025197 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.025250 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":471,"threads_started":1}
I20260924 04:03:10.025499 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.019117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.98.3:37231)
0924 04:03:10.019888 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.019921 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.020609 (+ 688us) server_negotiation.cc:407] Connection header received
0924 04:03:10.020990 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.020995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.021299 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.021761 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:10.021897 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.022876 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.023837 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.024693 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.024797 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.024967 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.025252 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.025356 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":168,"threads_started":1}
I20260924 04:03:10.026310 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.018531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35825 (local address 127.0.0.1:45164)
0924 04:03:10.019148 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.019203 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.019326 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.022551 (+ 3225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.022554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.022578 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:10.022668 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.022673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.023977 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.023980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.024952 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.024960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.025962 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.025965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.026056 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.026078 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.026120 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.026181 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":82,"threads_started":1}
I20260924 04:03:10.026336 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.019795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.0.98.1:33265)
0924 04:03:10.020115 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.020135 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.020159 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:03:10.020276 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.020282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.020509 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.021007 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:10.021142 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.022251 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.023886 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.024693 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.025979 (+ 1286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.026102 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.026141 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.026210 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260924 04:03:10.026860 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.018928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.0.98.2:35825)
0924 04:03:10.022164 (+ 3236us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.022172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.022190 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:03:10.022249 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.022253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.022412 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.022565 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:10.022756 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.023836 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.025065 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.025833 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.026488 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.026639 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.026685 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.026757 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1674,"thread_start_us":106,"threads_started":1}
I20260924 04:03:10.028479 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca (DEFAULT_TABLE table=test-table [id=b4cc7080a03946a4925079668068d9b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:10.029474 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.029458 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca (DEFAULT_TABLE table=test-table [id=b4cc7080a03946a4925079668068d9b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:10.030316 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.035943 2638 tablet_bootstrap.cc:492] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Bootstrap starting.
I20260924 04:03:10.036648 2639 tablet_bootstrap.cc:492] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Bootstrap starting.
I20260924 04:03:10.038300 2638 tablet_bootstrap.cc:654] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.038285 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca (DEFAULT_TABLE table=test-table [id=b4cc7080a03946a4925079668068d9b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:10.038790 2639 tablet_bootstrap.cc:654] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.039268 2638 log.cc:824] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:10.039402 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.039536 2639 log.cc:824] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:10.041400 2639 tablet_bootstrap.cc:492] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: No bootstrap required, opened a new log
I20260924 04:03:10.041400 2638 tablet_bootstrap.cc:492] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: No bootstrap required, opened a new log
I20260924 04:03:10.041590 2638 ts_tablet_manager.cc:1397] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:03:10.041590 2639 ts_tablet_manager.cc:1397] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:03:10.045502 2638 raft_consensus.cc:348] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.045502 2639 raft_consensus.cc:348] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.045786 2639 raft_consensus.cc:374] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.045869 2639 raft_consensus.cc:729] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26264b2caebf4120abcceb6211ef6530, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.045965 2638 raft_consensus.cc:374] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.046070 2638 raft_consensus.cc:729] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34688313822c433db985899599b188d7, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.046406 2639 consensus_queue.cc:260] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.046442 2638 consensus_queue.cc:260] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.046689 2642 tablet_bootstrap.cc:492] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Bootstrap starting.
I20260924 04:03:10.047487 2639 ts_tablet_manager.cc:1428] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:03:10.047487 2638 ts_tablet_manager.cc:1428] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:03:10.047863 2615 heartbeater.cc:503] Master 127.0.98.62:38615 was elected leader, sending a full tablet report...
I20260924 04:03:10.047869 2475 heartbeater.cc:503] Master 127.0.98.62:38615 was elected leader, sending a full tablet report...
I20260924 04:03:10.049001 2642 tablet_bootstrap.cc:654] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.050297 2642 log.cc:824] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:10.052490 2642 tablet_bootstrap.cc:492] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: No bootstrap required, opened a new log
I20260924 04:03:10.052678 2642 ts_tablet_manager.cc:1397] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:03:10.055771 2642 raft_consensus.cc:348] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.056058 2642 raft_consensus.cc:374] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.056115 2642 raft_consensus.cc:729] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b186b40397ec41faaa08dd73495aecbb, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.056545 2642 consensus_queue.cc:260] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.057155 2335 heartbeater.cc:503] Master 127.0.98.62:38615 was elected leader, sending a full tablet report...
I20260924 04:03:10.057498 2642 ts_tablet_manager.cc:1428] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:03:10.185072 2644 raft_consensus.cc:482] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:10.185307 2644 raft_consensus.cc:504] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.186435 2644 leader_election.cc:288] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825), b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265)
I20260924 04:03:10.189844 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.186797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51461 (local address 127.0.98.2:35825)
0924 04:03:10.186927 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.186932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.186948 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:03:10.187008 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.187011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.187058 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.187132 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:10.187429 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.187964 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.188953 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.189352 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.189442 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.189501 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.189660 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.189726 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:03:10.189844 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.186692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35825 (local address 127.0.98.3:51461)
0924 04:03:10.186809 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.186822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.186929 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.187169 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.187172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.187197 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:10.187315 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.187321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.188065 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.188068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.188837 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.188845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.189459 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.189462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.189556 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.189573 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.189635 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.189718 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:03:10.190469 2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5ad344c1e64ab1b2cfcd49952eb4ca" candidate_uuid: "34688313822c433db985899599b188d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26264b2caebf4120abcceb6211ef6530" is_pre_election: true
I20260924 04:03:10.190799 2429 raft_consensus.cc:2454] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34688313822c433db985899599b188d7 in term 0.
I20260924 04:03:10.190860 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.186967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33265 (local address 127.0.98.3:53061)
0924 04:03:10.187358 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.187368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.187457 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.187671 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.187674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.187688 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:10.187798 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.187803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.188532 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.188535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.189683 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.189699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.190458 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.190462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.190564 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.190581 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.190676 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.190746 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":135,"threads_started":1}
I20260924 04:03:10.190912 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.187140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53061 (local address 127.0.98.1:33265)
0924 04:03:10.187265 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.187268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.187441 (+ 173us) server_negotiation.cc:407] Connection header received
0924 04:03:10.187534 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.187537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.187584 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.187650 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:10.187891 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.188423 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.189806 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.190333 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.190425 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.190484 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.190714 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.190799 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:03:10.191282 2501 leader_election.cc:302] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26264b2caebf4120abcceb6211ef6530, 34688313822c433db985899599b188d7; no voters:
I20260924 04:03:10.191481 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5ad344c1e64ab1b2cfcd49952eb4ca" candidate_uuid: "34688313822c433db985899599b188d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b186b40397ec41faaa08dd73495aecbb" is_pre_election: true
I20260924 04:03:10.191542 2644 raft_consensus.cc:2781] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:10.191613 2644 raft_consensus.cc:482] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:10.191643 2644 raft_consensus.cc:3037] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.191797 2289 raft_consensus.cc:2454] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34688313822c433db985899599b188d7 in term 0.
I20260924 04:03:10.192862 2644 raft_consensus.cc:504] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.193197 2644 leader_election.cc:288] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [CANDIDATE]: Term 1 election: Requested vote from peers 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825), b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265)
I20260924 04:03:10.193511 2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5ad344c1e64ab1b2cfcd49952eb4ca" candidate_uuid: "34688313822c433db985899599b188d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26264b2caebf4120abcceb6211ef6530"
I20260924 04:03:10.193552 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5ad344c1e64ab1b2cfcd49952eb4ca" candidate_uuid: "34688313822c433db985899599b188d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b186b40397ec41faaa08dd73495aecbb"
I20260924 04:03:10.193640 2429 raft_consensus.cc:3037] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.193661 2289 raft_consensus.cc:3037] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.194635 2429 raft_consensus.cc:2454] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34688313822c433db985899599b188d7 in term 1.
I20260924 04:03:10.194722 2289 raft_consensus.cc:2454] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34688313822c433db985899599b188d7 in term 1.
I20260924 04:03:10.194916 2501 leader_election.cc:302] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26264b2caebf4120abcceb6211ef6530, 34688313822c433db985899599b188d7; no voters:
I20260924 04:03:10.195078 2644 raft_consensus.cc:2781] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:10.195305 2644 raft_consensus.cc:686] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 1 LEADER]: Becoming Leader. State: Replica: 34688313822c433db985899599b188d7, State: Running, Role: LEADER
I20260924 04:03:10.195647 2644 consensus_queue.cc:237] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.198743 2131 catalog_manager.cc:5495] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34688313822c433db985899599b188d7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } health_report { overall_health: HEALTHY } } }
W20260924 04:03:10.208964 2616 tablet.cc:2369] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:03:10 dist-test-slave-zcsk krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260924 04:03:10.237685 2476 tablet.cc:2369] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:10.237847 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.229684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.98.62:38615)
0924 04:03:10.229805 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.229810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.230031 (+ 221us) server_negotiation.cc:407] Connection header received
0924 04:03:10.230152 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.230155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.230194 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.230260 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:10.230921 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.231422 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.232255 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.232455 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.232537 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.234711 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:10.234714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:10.234716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:10.234737 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:10.236219 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:10.236661 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:10.236665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:10.236666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:10.236733 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:10.236952 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:10.236955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:10.236956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:10.237160 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:10.237243 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.237548 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.237705 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":43}
I20260924 04:03:10.237847 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.229655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38615 (local address 127.0.0.1:53042)
0924 04:03:10.229948 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.229960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.230073 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.230291 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.230293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.230717 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:10.230811 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.230817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.231515 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.231518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.232145 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.232157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.232549 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.232552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.232669 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.233217 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:10.233229 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:10.234575 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:10.236325 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:10.236335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:10.236341 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:10.236564 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:10.236806 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:10.236809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:10.236811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:10.236879 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:10.237258 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:10.237265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:10.237374 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.237469 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.237654 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":30,"thread_start_us":123,"threads_started":1}
I20260924 04:03:10.242223 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53042:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:03:10.242974 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:10.247539 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 623afd0194814db283f904e3eae8e051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c1b5032477c48adbba0058f4ab1eeec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:10.247493 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 623afd0194814db283f904e3eae8e051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c1b5032477c48adbba0058f4ab1eeec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:10.247498 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 623afd0194814db283f904e3eae8e051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c1b5032477c48adbba0058f4ab1eeec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:10.248282 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623afd0194814db283f904e3eae8e051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.248281 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623afd0194814db283f904e3eae8e051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.248281 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623afd0194814db283f904e3eae8e051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.250713 2642 tablet_bootstrap.cc:492] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: Bootstrap starting.
I20260924 04:03:10.251060 2638 tablet_bootstrap.cc:492] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: Bootstrap starting.
I20260924 04:03:10.251623 2642 tablet_bootstrap.cc:654] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.251855 2638 tablet_bootstrap.cc:654] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.252733 2639 tablet_bootstrap.cc:492] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: Bootstrap starting.
I20260924 04:03:10.252853 2642 tablet_bootstrap.cc:492] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: No bootstrap required, opened a new log
I20260924 04:03:10.252933 2642 ts_tablet_manager.cc:1397] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:10.253157 2638 tablet_bootstrap.cc:492] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: No bootstrap required, opened a new log
I20260924 04:03:10.253235 2638 ts_tablet_manager.cc:1397] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:10.253342 2642 raft_consensus.cc:348] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.253432 2642 raft_consensus.cc:374] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.253451 2642 raft_consensus.cc:729] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b186b40397ec41faaa08dd73495aecbb, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.253556 2642 consensus_queue.cc:260] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.253659 2639 tablet_bootstrap.cc:654] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.253667 2638 raft_consensus.cc:348] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.253763 2638 raft_consensus.cc:374] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.253793 2638 raft_consensus.cc:729] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34688313822c433db985899599b188d7, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.253804 2642 ts_tablet_manager.cc:1428] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:03:10.253824 2646 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.253947 2646 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:10.253913 2638 consensus_queue.cc:260] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.254158 2638 ts_tablet_manager.cc:1428] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:10.254369 2644 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.254485 2644 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:10.255461 2639 tablet_bootstrap.cc:492] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: No bootstrap required, opened a new log
I20260924 04:03:10.255563 2639 ts_tablet_manager.cc:1397] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:10.256047 2639 raft_consensus.cc:348] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.256168 2639 raft_consensus.cc:374] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.256193 2639 raft_consensus.cc:729] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26264b2caebf4120abcceb6211ef6530, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.256349 2639 consensus_queue.cc:260] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.256610 2639 ts_tablet_manager.cc:1428] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:10.256606 2643 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.256783 2643 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 04:03:10.283087 2336 tablet.cc:2369] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:10.363842 2644 raft_consensus.cc:482] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:10.364014 2644 raft_consensus.cc:504] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.364431 2644 leader_election.cc:288] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265), 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825)
I20260924 04:03:10.364926 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623afd0194814db283f904e3eae8e051" candidate_uuid: "34688313822c433db985899599b188d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b186b40397ec41faaa08dd73495aecbb" is_pre_election: true
I20260924 04:03:10.364977 2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623afd0194814db283f904e3eae8e051" candidate_uuid: "34688313822c433db985899599b188d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26264b2caebf4120abcceb6211ef6530" is_pre_election: true
I20260924 04:03:10.365074 2289 raft_consensus.cc:2454] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34688313822c433db985899599b188d7 in term 0.
I20260924 04:03:10.365108 2429 raft_consensus.cc:2454] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34688313822c433db985899599b188d7 in term 0.
I20260924 04:03:10.365368 2501 leader_election.cc:302] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34688313822c433db985899599b188d7, b186b40397ec41faaa08dd73495aecbb; no voters:
I20260924 04:03:10.365527 2644 raft_consensus.cc:2781] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:10.365583 2644 raft_consensus.cc:482] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:10.365608 2644 raft_consensus.cc:3037] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.366566 2644 raft_consensus.cc:504] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.366879 2644 leader_election.cc:288] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [CANDIDATE]: Term 1 election: Requested vote from peers b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265), 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825)
I20260924 04:03:10.367234 2429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623afd0194814db283f904e3eae8e051" candidate_uuid: "34688313822c433db985899599b188d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26264b2caebf4120abcceb6211ef6530"
I20260924 04:03:10.367249 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623afd0194814db283f904e3eae8e051" candidate_uuid: "34688313822c433db985899599b188d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b186b40397ec41faaa08dd73495aecbb"
I20260924 04:03:10.367343 2289 raft_consensus.cc:3037] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.367347 2429 raft_consensus.cc:3037] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.368224 2289 raft_consensus.cc:2454] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34688313822c433db985899599b188d7 in term 1.
I20260924 04:03:10.368302 2429 raft_consensus.cc:2454] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34688313822c433db985899599b188d7 in term 1.
I20260924 04:03:10.368508 2501 leader_election.cc:302] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34688313822c433db985899599b188d7, b186b40397ec41faaa08dd73495aecbb; no voters:
I20260924 04:03:10.368667 2644 raft_consensus.cc:2781] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:10.368770 2644 raft_consensus.cc:686] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [term 1 LEADER]: Becoming Leader. State: Replica: 34688313822c433db985899599b188d7, State: Running, Role: LEADER
I20260924 04:03:10.368901 2644 consensus_queue.cc:237] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.369527 2648 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: TxnStatusTablet state changed. Reason: New leader 34688313822c433db985899599b188d7. Latest consensus state: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.369635 2648 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:10.369941 2660 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:03:10.370164 2660 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:03:10.370576 2130 catalog_manager.cc:5495] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34688313822c433db985899599b188d7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } health_report { overall_health: HEALTHY } } }
I20260924 04:03:10.382189 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.378935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.0.98.3:37231)
0924 04:03:10.379053 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.379057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.379067 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:03:10.379120 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.379122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.379177 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.379259 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:10.379482 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.380024 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.380823 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.381051 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.381141 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.381394 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.381800 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.381905 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.381971 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.382025 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:03:10.382169 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.378563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37231 (local address 127.0.0.1:51896)
0924 04:03:10.378703 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.378717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.378804 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.379288 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.379291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.379299 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:10.379391 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.379396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.380124 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.380128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.380725 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.380733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.381140 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.381144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.381232 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.381273 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.381885 (+ 612us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.381900 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.381962 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.382046 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:03:10.387060 2429 raft_consensus.cc:1261] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Refusing update from remote peer 34688313822c433db985899599b188d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:10.387163 2289 raft_consensus.cc:1261] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Refusing update from remote peer 34688313822c433db985899599b188d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:10.387739 2648 consensus_queue.cc:1035] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:10.387955 2644 consensus_queue.cc:1035] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:10.391934 2643 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: TxnStatusTablet state changed. Reason: New leader 34688313822c433db985899599b188d7. Latest consensus state: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.392053 2643 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:10.392406 2646 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: TxnStatusTablet state changed. Reason: New leader 34688313822c433db985899599b188d7. Latest consensus state: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.392870 2646 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:10.395604 2648 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.395733 2648 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:10.395949 2646 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.396051 2646 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P b186b40397ec41faaa08dd73495aecbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:10.396328 2643 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.396426 2643 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P 26264b2caebf4120abcceb6211ef6530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:10.396929 2664 mvcc.cc:205] Tried to move back new op lower bound from 7332751730219282432 to 7332751730152353792. Current Snapshot: MvccSnapshot[applied={T|T < 7332751730219282432}]
I20260924 04:03:10.395855 2648 tablet_replica.cc:439] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34688313822c433db985899599b188d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } } }
I20260924 04:03:10.397914 2648 tablet_replica.cc:442] T 623afd0194814db283f904e3eae8e051 P 34688313822c433db985899599b188d7: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:10.401353 2665 mvcc.cc:205] Tried to move back new op lower bound from 7332751730219282432 to 7332751730152353792. Current Snapshot: MvccSnapshot[applied={T|T < 7332751730219282432}]
I20260924 04:03:10.408325 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.404955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.0.98.3:37231)
0924 04:03:10.405071 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.405074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.405086 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:03:10.405210 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.405212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.405251 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.405308 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:10.405558 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.406051 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.407267 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.407485 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.407569 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.407864 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.407976 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.408054 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.408153 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.408198 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:10.418926 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.415749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38257 (local address 127.0.98.3:37231)
0924 04:03:10.415846 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.415850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.415863 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:10.415984 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.415987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.416025 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.416082 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:10.416310 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.416782 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.417976 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.418196 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.418279 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.418565 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.418622 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.418668 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.418761 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.418798 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:03:10.418967 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.415659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37231 (local address 127.0.98.3:38257)
0924 04:03:10.415783 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.415797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.415883 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.416099 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.416102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.416108 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:10.416222 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.416227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.416867 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.416870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.417855 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.417865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.418291 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.418294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.418386 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.418431 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.418687 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.418693 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.418745 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.418813 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:03:10.425000 2429 raft_consensus.cc:1261] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Refusing update from remote peer 34688313822c433db985899599b188d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:10.425030 2289 raft_consensus.cc:1261] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Refusing update from remote peer 34688313822c433db985899599b188d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:10.425383 2668 consensus_queue.cc:1035] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 04:03:10.426064 2648 consensus_queue.cc:1035] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:10.430346 2663 mvcc.cc:205] Tried to move back new op lower bound from 7332751730377981952 to 7332751729444364288. Current Snapshot: MvccSnapshot[applied={T|T < 7332751730377981952}]
I20260924 04:03:10.463696 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:10.463867 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:10.466202 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: Sending DeleteTablet for 3 replicas of tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca
I20260924 04:03:10.466919 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260924 04:03:10.466939 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45164
I20260924 04:03:10.467515 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:03:10.467698 2681 tablet_replica.cc:330] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: stopping tablet replica
I20260924 04:03:10.467660 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34322
W20260924 04:03:10.467952 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:10.468041 2681 raft_consensus.cc:2229] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:10.468417 2681 raft_consensus.cc:2258] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:10.468739 2682 tablet_replica.cc:330] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: stopping tablet replica
I20260924 04:03:10.469026 2682 raft_consensus.cc:2229] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:10.469233 2682 raft_consensus.cc:2258] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:10.469916 2681 ts_tablet_manager.cc:1905] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:10.472759 2682 ts_tablet_manager.cc:1905] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:10.472765 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 7d16a1ad11764f87a79746f0555a339d (DEFAULT_TABLE table=test-table [id=fdaeb74b71004c66b24b14f448b2a8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:10.473078 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d16a1ad11764f87a79746f0555a339d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.473811 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 7d16a1ad11764f87a79746f0555a339d (DEFAULT_TABLE table=test-table [id=fdaeb74b71004c66b24b14f448b2a8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:10.474071 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d16a1ad11764f87a79746f0555a339d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.475582 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 7d16a1ad11764f87a79746f0555a339d (DEFAULT_TABLE table=test-table [id=fdaeb74b71004c66b24b14f448b2a8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:10.475858 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d16a1ad11764f87a79746f0555a339d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:10.476329 2682 ts_tablet_manager.cc:1918] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:10.476477 2682 log.cc:1196] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/5a5ad344c1e64ab1b2cfcd49952eb4ca
I20260924 04:03:10.476853 2682 ts_tablet_manager.cc:1939] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 26264b2caebf4120abcceb6211ef6530: Deleting consensus metadata
I20260924 04:03:10.477355 2638 tablet_bootstrap.cc:492] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: Bootstrap starting.
I20260924 04:03:10.478166 2639 tablet_bootstrap.cc:492] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: Bootstrap starting.
I20260924 04:03:10.478188 2638 tablet_bootstrap.cc:654] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.479005 2639 tablet_bootstrap.cc:654] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.479439 2638 tablet_bootstrap.cc:492] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: No bootstrap required, opened a new log
I20260924 04:03:10.479534 2638 ts_tablet_manager.cc:1397] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:10.479746 2642 tablet_bootstrap.cc:492] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: Bootstrap starting.
I20260924 04:03:10.479991 2638 raft_consensus.cc:348] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.480110 2638 raft_consensus.cc:374] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.480149 2638 raft_consensus.cc:729] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34688313822c433db985899599b188d7, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.480273 2638 consensus_queue.cc:260] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.480516 2638 ts_tablet_manager.cc:1428] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:10.480674 2642 tablet_bootstrap.cc:654] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:10.481319 2117 catalog_manager.cc:4841] TS 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825): tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca (table test-table [id=b4cc7080a03946a4925079668068d9b7]) successfully deleted
I20260924 04:03:10.481544 2639 tablet_bootstrap.cc:492] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: No bootstrap required, opened a new log
I20260924 04:03:10.481616 2639 ts_tablet_manager.cc:1397] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:03:10.482087 2639 raft_consensus.cc:348] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.482200 2639 raft_consensus.cc:374] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.482241 2639 raft_consensus.cc:729] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26264b2caebf4120abcceb6211ef6530, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.482362 2639 consensus_queue.cc:260] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.482625 2639 ts_tablet_manager.cc:1428] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:10.483210 2642 tablet_bootstrap.cc:492] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: No bootstrap required, opened a new log
I20260924 04:03:10.483294 2642 ts_tablet_manager.cc:1397] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:03:10.483636 2642 raft_consensus.cc:348] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.483724 2642 raft_consensus.cc:374] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:10.483744 2642 raft_consensus.cc:729] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b186b40397ec41faaa08dd73495aecbb, State: Initialized, Role: FOLLOWER
I20260924 04:03:10.483858 2642 consensus_queue.cc:260] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.484125 2642 ts_tablet_manager.cc:1428] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:10.486155 2681 ts_tablet_manager.cc:1918] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:10.486268 2681 log.cc:1196] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/5a5ad344c1e64ab1b2cfcd49952eb4ca
I20260924 04:03:10.486564 2681 ts_tablet_manager.cc:1939] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P 34688313822c433db985899599b188d7: Deleting consensus metadata
I20260924 04:03:10.487644 2683 tablet_replica.cc:330] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: stopping tablet replica
I20260924 04:03:10.487603 2119 catalog_manager.cc:4841] TS 34688313822c433db985899599b188d7 (127.0.98.3:37231): tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca (table test-table [id=b4cc7080a03946a4925079668068d9b7]) successfully deleted
I20260924 04:03:10.487982 2683 raft_consensus.cc:2229] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:10.488197 2683 raft_consensus.cc:2258] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:10.489853 2683 ts_tablet_manager.cc:1905] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:10.492349 2683 ts_tablet_manager.cc:1918] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:10.492435 2683 log.cc:1196] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/5a5ad344c1e64ab1b2cfcd49952eb4ca
I20260924 04:03:10.492687 2683 ts_tablet_manager.cc:1939] T 5a5ad344c1e64ab1b2cfcd49952eb4ca P b186b40397ec41faaa08dd73495aecbb: Deleting consensus metadata
I20260924 04:03:10.493877 2117 catalog_manager.cc:4841] TS b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265): tablet 5a5ad344c1e64ab1b2cfcd49952eb4ca (table test-table [id=b4cc7080a03946a4925079668068d9b7]) successfully deleted
I20260924 04:03:10.655117 2643 raft_consensus.cc:482] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:10.655267 2643 raft_consensus.cc:504] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.656463 2643 leader_election.cc:288] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265), 34688313822c433db985899599b188d7 (127.0.98.3:37231)
I20260924 04:03:10.661067 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.656832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52481 (local address 127.0.98.1:33265)
0924 04:03:10.656949 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.656954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.657119 (+ 165us) server_negotiation.cc:407] Connection header received
0924 04:03:10.657336 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.657339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.657380 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.657441 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:10.657738 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.658509 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.660031 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.660446 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.660543 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.660609 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.660838 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.660926 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:03:10.661154 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.657183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42209 (local address 127.0.98.3:37231)
0924 04:03:10.657307 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.657312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.657327 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:03:10.657380 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.657383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.657434 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.657516 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:10.657798 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.658300 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.660001 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.660638 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.660745 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.660808 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.660947 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.661020 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:03:10.661432 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.656750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37231 (local address 127.0.98.2:42209)
0924 04:03:10.657103 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.657115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.657206 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.657558 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.657561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.657575 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:10.657656 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.657662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.658417 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.658421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.659842 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.659850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.660750 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.660753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.660841 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.660859 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.660922 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.660992 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":54,"spinlock_wait_cycles":13696,"thread_start_us":108,"threads_started":1}
I20260924 04:03:10.661551 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.656724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33265 (local address 127.0.98.2:52481)
0924 04:03:10.657030 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.657043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.657204 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.657464 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.657467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.657485 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:10.657604 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.657610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.658620 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.658623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.659836 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.659850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.660582 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.660586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.660690 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.660714 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.661306 (+ 592us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.661379 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":51,"thread_start_us":74,"threads_started":1}
I20260924 04:03:10.662031 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d16a1ad11764f87a79746f0555a339d" candidate_uuid: "26264b2caebf4120abcceb6211ef6530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b186b40397ec41faaa08dd73495aecbb" is_pre_election: true
I20260924 04:03:10.662204 2289 raft_consensus.cc:2454] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26264b2caebf4120abcceb6211ef6530 in term 0.
I20260924 04:03:10.662549 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d16a1ad11764f87a79746f0555a339d" candidate_uuid: "26264b2caebf4120abcceb6211ef6530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34688313822c433db985899599b188d7" is_pre_election: true
I20260924 04:03:10.662559 2361 leader_election.cc:302] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26264b2caebf4120abcceb6211ef6530, b186b40397ec41faaa08dd73495aecbb; no voters:
I20260924 04:03:10.662781 2569 raft_consensus.cc:2454] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26264b2caebf4120abcceb6211ef6530 in term 0.
I20260924 04:03:10.662850 2643 raft_consensus.cc:2781] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:10.662904 2643 raft_consensus.cc:482] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:10.662930 2643 raft_consensus.cc:3037] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.664201 2643 raft_consensus.cc:504] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.664557 2643 leader_election.cc:288] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [CANDIDATE]: Term 1 election: Requested vote from peers b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265), 34688313822c433db985899599b188d7 (127.0.98.3:37231)
I20260924 04:03:10.664892 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d16a1ad11764f87a79746f0555a339d" candidate_uuid: "26264b2caebf4120abcceb6211ef6530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b186b40397ec41faaa08dd73495aecbb"
I20260924 04:03:10.665014 2289 raft_consensus.cc:3037] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.666110 2289 raft_consensus.cc:2454] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26264b2caebf4120abcceb6211ef6530 in term 1.
I20260924 04:03:10.666363 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d16a1ad11764f87a79746f0555a339d" candidate_uuid: "26264b2caebf4120abcceb6211ef6530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34688313822c433db985899599b188d7"
I20260924 04:03:10.666385 2361 leader_election.cc:302] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26264b2caebf4120abcceb6211ef6530, b186b40397ec41faaa08dd73495aecbb; no voters:
I20260924 04:03:10.666479 2569 raft_consensus.cc:3037] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:10.666555 2643 raft_consensus.cc:2781] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:10.666749 2643 raft_consensus.cc:686] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 1 LEADER]: Becoming Leader. State: Replica: 26264b2caebf4120abcceb6211ef6530, State: Running, Role: LEADER
I20260924 04:03:10.667028 2643 consensus_queue.cc:237] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:10.667589 2569 raft_consensus.cc:2454] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26264b2caebf4120abcceb6211ef6530 in term 1.
I20260924 04:03:10.669484 2130 catalog_manager.cc:5495] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26264b2caebf4120abcceb6211ef6530 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "26264b2caebf4120abcceb6211ef6530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } health_report { overall_health: UNKNOWN } } }
I20260924 04:03:10.723178 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.719584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.0.98.2:35825)
0924 04:03:10.719832 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.719835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.719846 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:03:10.719948 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.719951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.720026 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.720138 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:10.720365 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.720880 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.721692 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.721924 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.722018 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.722278 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.722767 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.722873 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.722947 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.723012 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260924 04:03:10.733664 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.730062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37231 (local address 127.0.98.2:34919)
0924 04:03:10.730382 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.730395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.730496 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.730731 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.730735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.730742 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:10.730861 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.730866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.731536 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.731539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.732509 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.732518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.732932 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.732936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.733027 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.733068 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.733338 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.733344 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.733410 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.733487 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":124,"threads_started":1}
I20260924 04:03:10.733714 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.730204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34919 (local address 127.0.98.3:37231)
0924 04:03:10.730331 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.730336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.730459 (+ 123us) server_negotiation.cc:407] Connection header received
0924 04:03:10.730597 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.730599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.730653 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.730713 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:10.730956 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.731440 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.732627 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.732835 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.732920 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.733194 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.733266 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.733322 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.733493 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.733557 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:03:10.740119 2289 raft_consensus.cc:1261] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Refusing update from remote peer 26264b2caebf4120abcceb6211ef6530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:10.740329 2569 raft_consensus.cc:1261] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Refusing update from remote peer 26264b2caebf4120abcceb6211ef6530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:10.740572 2689 consensus_queue.cc:1035] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:10.740772 2643 consensus_queue.cc:1035] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:10.759430 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.756208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35825 (local address 127.0.98.3:51291)
0924 04:03:10.756329 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:10.756344 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:10.756417 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:10.756809 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:10.756812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:10.756821 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:10.756927 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.756934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.757679 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.757682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.758250 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:10.758257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.758740 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.758743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:10.758827 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.758860 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.759157 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.759162 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:03:10.759222 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:03:10.759306 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 04:03:10.759425 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:10.756365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51291 (local address 127.0.98.2:35825)
0924 04:03:10.756492 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:10.756497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:10.756512 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:03:10.756558 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:10.756561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:10.756615 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:10.756691 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:10.757043 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.757551 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.758351 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:10.758613 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:10.758714 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:10.758969 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:10.759047 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:10.759126 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:10.759240 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:03:10.759298 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 04:03:10.760316 2664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:03:12.317118 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:12.317262 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:12.319437 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: Sending DeleteTablet for 3 replicas of tablet 7d16a1ad11764f87a79746f0555a339d
I20260924 04:03:12.319938 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d16a1ad11764f87a79746f0555a339d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34322
I20260924 04:03:12.320025 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d16a1ad11764f87a79746f0555a339d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260924 04:03:12.320271 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d16a1ad11764f87a79746f0555a339d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45164
I20260924 04:03:12.320322 2720 tablet_replica.cc:330] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: stopping tablet replica
I20260924 04:03:12.320446 2720 raft_consensus.cc:2229] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:12.320554 2720 raft_consensus.cc:2258] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:12.321192 2720 ts_tablet_manager.cc:1905] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:12.322702 2722 tablet_replica.cc:330] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: stopping tablet replica
I20260924 04:03:12.322875 2722 raft_consensus.cc:2229] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:12.323197 2722 raft_consensus.cc:2258] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:12.323441 2721 tablet_replica.cc:330] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: stopping tablet replica
I20260924 04:03:12.323578 2721 raft_consensus.cc:2229] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:12.323701 2721 raft_consensus.cc:2258] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:12.323832 2720 ts_tablet_manager.cc:1918] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:03:12.323899 2720 log.cc:1196] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/7d16a1ad11764f87a79746f0555a339d
I20260924 04:03:12.324183 2720 ts_tablet_manager.cc:1939] T 7d16a1ad11764f87a79746f0555a339d P b186b40397ec41faaa08dd73495aecbb: Deleting consensus metadata
I20260924 04:03:12.324179 2722 ts_tablet_manager.cc:1905] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:12.324295 2721 ts_tablet_manager.cc:1905] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:12.324826 2117 catalog_manager.cc:4841] TS b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265): tablet 7d16a1ad11764f87a79746f0555a339d (table test-table [id=fdaeb74b71004c66b24b14f448b2a8d7]) successfully deleted
I20260924 04:03:12.326532 2721 ts_tablet_manager.cc:1918] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:03:12.326601 2721 log.cc:1196] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/7d16a1ad11764f87a79746f0555a339d
I20260924 04:03:12.326802 2721 ts_tablet_manager.cc:1939] T 7d16a1ad11764f87a79746f0555a339d P 34688313822c433db985899599b188d7: Deleting consensus metadata
I20260924 04:03:12.326973 2722 ts_tablet_manager.cc:1918] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:03:12.327052 2722 log.cc:1196] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/7d16a1ad11764f87a79746f0555a339d
I20260924 04:03:12.327306 2722 ts_tablet_manager.cc:1939] T 7d16a1ad11764f87a79746f0555a339d P 26264b2caebf4120abcceb6211ef6530: Deleting consensus metadata
I20260924 04:03:12.327402 2119 catalog_manager.cc:4841] TS 34688313822c433db985899599b188d7 (127.0.98.3:37231): tablet 7d16a1ad11764f87a79746f0555a339d (table test-table [id=fdaeb74b71004c66b24b14f448b2a8d7]) successfully deleted
I20260924 04:03:12.327998 2117 catalog_manager.cc:4841] TS 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825): tablet 7d16a1ad11764f87a79746f0555a339d (table test-table [id=fdaeb74b71004c66b24b14f448b2a8d7]) successfully deleted
Sep 24 04:03:12 dist-test-slave-zcsk krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222592, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:03:12 dist-test-slave-zcsk krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222592, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:12.346215 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.338130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.0.98.62:38615)
0924 04:03:12.338372 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:12.338378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:12.338392 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:12.338512 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:12.338515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:12.338556 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:12.338637 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:12.339244 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.339785 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.340625 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.340847 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.340922 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.343158 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:12.343162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:12.343165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:12.343192 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:12.344580 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:12.345020 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:12.345024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:12.345027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:12.345111 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:12.345364 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:12.345367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:12.345369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:12.345570 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:12.345694 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:12.345922 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 04:03:12.346052 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260924 04:03:12.348663 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:12.349051 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:12.353502 2409 tablet_service.cc:1467] Processing CreateTablet for tablet c0d33d4c787148b28a5c5c8ad4c77323 (DEFAULT_TABLE table=test-table [id=fddc66c7c6234753ae96ec76c3658cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:12.353528 2269 tablet_service.cc:1467] Processing CreateTablet for tablet c0d33d4c787148b28a5c5c8ad4c77323 (DEFAULT_TABLE table=test-table [id=fddc66c7c6234753ae96ec76c3658cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:12.353602 2532 tablet_service.cc:1467] Processing CreateTablet for tablet c0d33d4c787148b28a5c5c8ad4c77323 (DEFAULT_TABLE table=test-table [id=fddc66c7c6234753ae96ec76c3658cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:12.353811 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d33d4c787148b28a5c5c8ad4c77323. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:12.353811 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d33d4c787148b28a5c5c8ad4c77323. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:12.353822 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d33d4c787148b28a5c5c8ad4c77323. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:12.355818 2731 tablet_bootstrap.cc:492] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: Bootstrap starting.
I20260924 04:03:12.356218 2732 tablet_bootstrap.cc:492] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: Bootstrap starting.
I20260924 04:03:12.356835 2731 tablet_bootstrap.cc:654] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:12.357016 2732 tablet_bootstrap.cc:654] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:12.357653 2733 tablet_bootstrap.cc:492] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: Bootstrap starting.
I20260924 04:03:12.358208 2731 tablet_bootstrap.cc:492] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: No bootstrap required, opened a new log
I20260924 04:03:12.358297 2731 ts_tablet_manager.cc:1397] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:12.358500 2733 tablet_bootstrap.cc:654] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:12.358750 2731 raft_consensus.cc:348] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.358870 2731 raft_consensus.cc:374] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:12.358911 2731 raft_consensus.cc:729] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b186b40397ec41faaa08dd73495aecbb, State: Initialized, Role: FOLLOWER
I20260924 04:03:12.359042 2731 consensus_queue.cc:260] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.359331 2732 tablet_bootstrap.cc:492] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: No bootstrap required, opened a new log
I20260924 04:03:12.359411 2731 ts_tablet_manager.cc:1428] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:12.359418 2732 ts_tablet_manager.cc:1397] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:03:12.359818 2732 raft_consensus.cc:348] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.359939 2732 raft_consensus.cc:374] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:12.359972 2732 raft_consensus.cc:729] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26264b2caebf4120abcceb6211ef6530, State: Initialized, Role: FOLLOWER
I20260924 04:03:12.360118 2732 consensus_queue.cc:260] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.360208 2733 tablet_bootstrap.cc:492] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: No bootstrap required, opened a new log
I20260924 04:03:12.360302 2733 ts_tablet_manager.cc:1397] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:12.360726 2732 ts_tablet_manager.cc:1428] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:03:12.360688 2733 raft_consensus.cc:348] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.360795 2733 raft_consensus.cc:374] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:12.360826 2733 raft_consensus.cc:729] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34688313822c433db985899599b188d7, State: Initialized, Role: FOLLOWER
I20260924 04:03:12.360939 2733 consensus_queue.cc:260] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.361289 2733 ts_tablet_manager.cc:1428] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:12.411629 2737 raft_consensus.cc:482] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:12.411760 2737 raft_consensus.cc:504] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.412815 2737 leader_election.cc:288] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825), 34688313822c433db985899599b188d7 (127.0.98.3:37231)
I20260924 04:03:12.416886 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.413155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49469 (local address 127.0.98.2:35825)
0924 04:03:12.413424 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:12.413433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:12.413510 (+ 77us) server_negotiation.cc:407] Connection header received
0924 04:03:12.413658 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:12.413661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:12.413708 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:12.413767 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:12.414112 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.414710 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.415890 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.416314 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.416401 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.416475 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:12.416674 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 04:03:12.416750 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260924 04:03:12.416886 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.413155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54163 (local address 127.0.98.3:37231)
0924 04:03:12.413432 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:12.413440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:12.413581 (+ 141us) server_negotiation.cc:407] Connection header received
0924 04:03:12.413693 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:12.413695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:12.413734 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:12.413810 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:12.414176 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.414748 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.415936 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.416331 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.416424 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.416487 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:12.416686 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 04:03:12.416755 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260924 04:03:12.416911 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.413041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35825 (local address 127.0.98.1:49469)
0924 04:03:12.413419 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:12.413434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:12.413543 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:12.413795 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:12.413797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:12.413881 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:12.414016 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:12.414021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.414809 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.414813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:12.415776 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:12.415784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.416429 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.416433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:12.416564 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.416585 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:03:12.416644 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 04:03:12.416730 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":57,"spinlock_wait_cycles":22784,"thread_start_us":145,"threads_started":1}
I20260924 04:03:12.416916 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.413056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37231 (local address 127.0.98.1:54163)
0924 04:03:12.413491 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:12.413504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:12.413602 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:12.413826 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:12.413829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:12.413881 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:12.414050 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:12.414055 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.414850 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.414852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:12.415796 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:12.415805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.416437 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.416441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:12.416573 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.416593 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:03:12.416657 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:03:12.416750 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":74,"spinlock_wait_cycles":9984,"thread_start_us":135,"threads_started":1}
I20260924 04:03:12.417399 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d33d4c787148b28a5c5c8ad4c77323" candidate_uuid: "b186b40397ec41faaa08dd73495aecbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26264b2caebf4120abcceb6211ef6530" is_pre_election: true
I20260924 04:03:12.417572 2428 raft_consensus.cc:2454] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b186b40397ec41faaa08dd73495aecbb in term 0.
I20260924 04:03:12.417961 2221 leader_election.cc:302] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26264b2caebf4120abcceb6211ef6530, b186b40397ec41faaa08dd73495aecbb; no voters:
I20260924 04:03:12.417980 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d33d4c787148b28a5c5c8ad4c77323" candidate_uuid: "b186b40397ec41faaa08dd73495aecbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34688313822c433db985899599b188d7" is_pre_election: true
I20260924 04:03:12.418108 2569 raft_consensus.cc:2454] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b186b40397ec41faaa08dd73495aecbb in term 0.
I20260924 04:03:12.418220 2737 raft_consensus.cc:2781] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:12.418280 2737 raft_consensus.cc:482] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:12.418329 2737 raft_consensus.cc:3037] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:12.419355 2737 raft_consensus.cc:504] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.419687 2737 leader_election.cc:288] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [CANDIDATE]: Term 1 election: Requested vote from peers 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825), 34688313822c433db985899599b188d7 (127.0.98.3:37231)
I20260924 04:03:12.420017 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d33d4c787148b28a5c5c8ad4c77323" candidate_uuid: "b186b40397ec41faaa08dd73495aecbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34688313822c433db985899599b188d7"
I20260924 04:03:12.420125 2569 raft_consensus.cc:3037] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:12.420995 2569 raft_consensus.cc:2454] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b186b40397ec41faaa08dd73495aecbb in term 1.
I20260924 04:03:12.421213 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d33d4c787148b28a5c5c8ad4c77323" candidate_uuid: "b186b40397ec41faaa08dd73495aecbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26264b2caebf4120abcceb6211ef6530"
I20260924 04:03:12.421252 2223 leader_election.cc:302] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34688313822c433db985899599b188d7, b186b40397ec41faaa08dd73495aecbb; no voters:
I20260924 04:03:12.421316 2428 raft_consensus.cc:3037] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:12.421394 2737 raft_consensus.cc:2781] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:12.421628 2737 raft_consensus.cc:686] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 1 LEADER]: Becoming Leader. State: Replica: b186b40397ec41faaa08dd73495aecbb, State: Running, Role: LEADER
I20260924 04:03:12.421862 2737 consensus_queue.cc:237] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.422422 2428 raft_consensus.cc:2454] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b186b40397ec41faaa08dd73495aecbb in term 1.
I20260924 04:03:12.424212 2131 catalog_manager.cc:5495] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb reported cstate change: term changed from 0 to 1, leader changed from <none> to b186b40397ec41faaa08dd73495aecbb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b186b40397ec41faaa08dd73495aecbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } health_report { overall_health: UNKNOWN } } }
I20260924 04:03:12.446693 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.442973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.0.98.1:33265)
0924 04:03:12.443201 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:12.443207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:12.443219 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:03:12.443296 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:12.443298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:12.443351 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:12.443483 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:12.443747 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.444351 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.445231 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.445486 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.445568 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.445881 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:12.446372 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:12.446458 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:12.446512 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:03:12.446561 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260924 04:03:12.456810 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.453612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37917 (local address 127.0.98.3:37231)
0924 04:03:12.453721 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:12.453725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:12.453738 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:12.453840 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:12.453877 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:12.453925 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:12.454003 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:12.454303 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.454829 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.455648 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.455864 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.456009 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.456308 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:12.456394 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:12.456470 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:12.456648 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:03:12.456700 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:03:12.456818 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.453379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37231 (local address 127.0.98.1:37917)
0924 04:03:12.453668 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:12.453680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:12.453760 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:12.454031 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:12.454034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:12.454043 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:12.454164 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:12.454169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.454917 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.454919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:12.455541 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:12.455548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.456018 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.456021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:12.456093 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.456139 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:12.456515 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:12.456522 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:03:12.456590 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:03:12.456667 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260924 04:03:12.463076 2569 raft_consensus.cc:1261] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Refusing update from remote peer b186b40397ec41faaa08dd73495aecbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:12.463254 2428 raft_consensus.cc:1261] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Refusing update from remote peer b186b40397ec41faaa08dd73495aecbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:12.463555 2742 consensus_queue.cc:1035] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:12.463716 2737 consensus_queue.cc:1035] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:12.479679 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.476681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54261 (local address 127.0.98.1:33265)
0924 04:03:12.476782 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:12.476786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:12.476809 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:03:12.476902 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:12.476904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:12.476942 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:12.476990 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:12.477165 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.477681 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.478661 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.478889 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.479004 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.479238 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:12.479323 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:12.479394 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:12.479484 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:03:12.479542 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:03:12.479691 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.476443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33265 (local address 127.0.98.3:54261)
0924 04:03:12.476711 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:12.476726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:12.476815 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:12.477005 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:12.477007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:12.477014 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:12.477104 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:12.477108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.477780 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.477783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:12.478531 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:12.478541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.478999 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.479003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:12.479092 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.479124 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:12.479405 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:12.479411 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:03:12.479464 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:03:12.479552 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
I20260924 04:03:12.502821 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:12.502995 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:12.505095 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: Sending DeleteTablet for 3 replicas of tablet c0d33d4c787148b28a5c5c8ad4c77323
I20260924 04:03:12.505699 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d33d4c787148b28a5c5c8ad4c77323 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260924 04:03:12.505699 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d33d4c787148b28a5c5c8ad4c77323 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45164
I20260924 04:03:12.505719 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d33d4c787148b28a5c5c8ad4c77323 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34322
I20260924 04:03:12.505885 2721 tablet_replica.cc:330] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: stopping tablet replica
I20260924 04:03:12.505904 2720 tablet_replica.cc:330] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: stopping tablet replica
I20260924 04:03:12.505908 2722 tablet_replica.cc:330] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: stopping tablet replica
I20260924 04:03:12.506003 2721 raft_consensus.cc:2229] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:12.506011 2720 raft_consensus.cc:2229] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:12.506016 2722 raft_consensus.cc:2229] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:12.506116 2722 raft_consensus.cc:2258] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:12.506116 2721 raft_consensus.cc:2258] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:12.506340 2720 raft_consensus.cc:2258] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:12.506933 2721 ts_tablet_manager.cc:1905] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:12.506877 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:12.507269 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:12.507296 2720 ts_tablet_manager.cc:1905] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:12.509307 2722 ts_tablet_manager.cc:1905] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:12.509630 2721 ts_tablet_manager.cc:1918] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:12.509693 2721 log.cc:1196] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/c0d33d4c787148b28a5c5c8ad4c77323
I20260924 04:03:12.509989 2720 ts_tablet_manager.cc:1918] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:12.510088 2720 log.cc:1196] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/c0d33d4c787148b28a5c5c8ad4c77323
I20260924 04:03:12.510324 2720 ts_tablet_manager.cc:1939] T c0d33d4c787148b28a5c5c8ad4c77323 P b186b40397ec41faaa08dd73495aecbb: Deleting consensus metadata
I20260924 04:03:12.510886 2721 ts_tablet_manager.cc:1939] T c0d33d4c787148b28a5c5c8ad4c77323 P 34688313822c433db985899599b188d7: Deleting consensus metadata
I20260924 04:03:12.511019 2117 catalog_manager.cc:4841] TS b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265): tablet c0d33d4c787148b28a5c5c8ad4c77323 (table test-table [id=fddc66c7c6234753ae96ec76c3658cf0]) successfully deleted
I20260924 04:03:12.511519 2119 catalog_manager.cc:4841] TS 34688313822c433db985899599b188d7 (127.0.98.3:37231): tablet c0d33d4c787148b28a5c5c8ad4c77323 (table test-table [id=fddc66c7c6234753ae96ec76c3658cf0]) successfully deleted
I20260924 04:03:12.511744 2722 ts_tablet_manager.cc:1918] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:12.511816 2722 log.cc:1196] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/c0d33d4c787148b28a5c5c8ad4c77323
I20260924 04:03:12.512064 2722 ts_tablet_manager.cc:1939] T c0d33d4c787148b28a5c5c8ad4c77323 P 26264b2caebf4120abcceb6211ef6530: Deleting consensus metadata
I20260924 04:03:12.512995 2117 catalog_manager.cc:4841] TS 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825): tablet c0d33d4c787148b28a5c5c8ad4c77323 (table test-table [id=fddc66c7c6234753ae96ec76c3658cf0]) successfully deleted
I20260924 04:03:12.514529 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 3b30fdb78b6947b29c6c0ada9703a6b3 (DEFAULT_TABLE table=test-table [id=c11534a7046d44ed8c5811c0c5593ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:12.514779 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b30fdb78b6947b29c6c0ada9703a6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:12.514778 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 3b30fdb78b6947b29c6c0ada9703a6b3 (DEFAULT_TABLE table=test-table [id=c11534a7046d44ed8c5811c0c5593ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:12.514997 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b30fdb78b6947b29c6c0ada9703a6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:12.516877 2733 tablet_bootstrap.cc:492] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: Bootstrap starting.
I20260924 04:03:12.517019 2732 tablet_bootstrap.cc:492] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: Bootstrap starting.
I20260924 04:03:12.517621 2733 tablet_bootstrap.cc:654] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:12.517786 2732 tablet_bootstrap.cc:654] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:12.518405 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 3b30fdb78b6947b29c6c0ada9703a6b3 (DEFAULT_TABLE table=test-table [id=c11534a7046d44ed8c5811c0c5593ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:12.518543 2733 tablet_bootstrap.cc:492] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: No bootstrap required, opened a new log
I20260924 04:03:12.518622 2733 ts_tablet_manager.cc:1397] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:12.518656 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b30fdb78b6947b29c6c0ada9703a6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:12.518986 2733 raft_consensus.cc:348] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.519083 2733 raft_consensus.cc:374] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:12.519111 2733 raft_consensus.cc:729] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34688313822c433db985899599b188d7, State: Initialized, Role: FOLLOWER
I20260924 04:03:12.519227 2733 consensus_queue.cc:260] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.519475 2733 ts_tablet_manager.cc:1428] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:12.520851 2731 tablet_bootstrap.cc:492] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb: Bootstrap starting.
I20260924 04:03:12.521739 2731 tablet_bootstrap.cc:654] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:12.522382 2732 tablet_bootstrap.cc:492] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: No bootstrap required, opened a new log
I20260924 04:03:12.522486 2732 ts_tablet_manager.cc:1397] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260924 04:03:12.522917 2732 raft_consensus.cc:348] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.523041 2732 raft_consensus.cc:374] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:12.523097 2732 raft_consensus.cc:729] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26264b2caebf4120abcceb6211ef6530, State: Initialized, Role: FOLLOWER
I20260924 04:03:12.523221 2732 consensus_queue.cc:260] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.523488 2732 ts_tablet_manager.cc:1428] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:03:12.525012 2731 tablet_bootstrap.cc:492] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb: No bootstrap required, opened a new log
I20260924 04:03:12.525120 2731 ts_tablet_manager.cc:1397] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:03:12.525591 2731 raft_consensus.cc:348] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.525696 2731 raft_consensus.cc:374] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:12.525719 2731 raft_consensus.cc:729] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b186b40397ec41faaa08dd73495aecbb, State: Initialized, Role: FOLLOWER
I20260924 04:03:12.525846 2731 consensus_queue.cc:260] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.526140 2731 ts_tablet_manager.cc:1428] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:12.685433 2711 raft_consensus.cc:482] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:12.685575 2711 raft_consensus.cc:504] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.685954 2711 leader_election.cc:288] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265), 34688313822c433db985899599b188d7 (127.0.98.3:37231)
I20260924 04:03:12.686435 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b30fdb78b6947b29c6c0ada9703a6b3" candidate_uuid: "26264b2caebf4120abcceb6211ef6530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34688313822c433db985899599b188d7" is_pre_election: true
I20260924 04:03:12.686455 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b30fdb78b6947b29c6c0ada9703a6b3" candidate_uuid: "26264b2caebf4120abcceb6211ef6530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b186b40397ec41faaa08dd73495aecbb" is_pre_election: true
I20260924 04:03:12.686569 2569 raft_consensus.cc:2454] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26264b2caebf4120abcceb6211ef6530 in term 0.
I20260924 04:03:12.686621 2288 raft_consensus.cc:2454] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26264b2caebf4120abcceb6211ef6530 in term 0.
I20260924 04:03:12.686923 2363 leader_election.cc:302] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26264b2caebf4120abcceb6211ef6530, 34688313822c433db985899599b188d7; no voters:
I20260924 04:03:12.687103 2711 raft_consensus.cc:2781] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:12.687160 2711 raft_consensus.cc:482] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:12.687207 2711 raft_consensus.cc:3037] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:12.688251 2711 raft_consensus.cc:504] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.688584 2711 leader_election.cc:288] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [CANDIDATE]: Term 1 election: Requested vote from peers b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265), 34688313822c433db985899599b188d7 (127.0.98.3:37231)
I20260924 04:03:12.688884 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b30fdb78b6947b29c6c0ada9703a6b3" candidate_uuid: "26264b2caebf4120abcceb6211ef6530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b186b40397ec41faaa08dd73495aecbb"
I20260924 04:03:12.689029 2288 raft_consensus.cc:3037] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:12.688993 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b30fdb78b6947b29c6c0ada9703a6b3" candidate_uuid: "26264b2caebf4120abcceb6211ef6530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34688313822c433db985899599b188d7"
I20260924 04:03:12.689117 2569 raft_consensus.cc:3037] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:12.689954 2288 raft_consensus.cc:2454] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26264b2caebf4120abcceb6211ef6530 in term 1.
I20260924 04:03:12.690026 2569 raft_consensus.cc:2454] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26264b2caebf4120abcceb6211ef6530 in term 1.
I20260924 04:03:12.690210 2361 leader_election.cc:302] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26264b2caebf4120abcceb6211ef6530, b186b40397ec41faaa08dd73495aecbb; no voters:
I20260924 04:03:12.690366 2711 raft_consensus.cc:2781] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:12.690469 2711 raft_consensus.cc:686] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 1 LEADER]: Becoming Leader. State: Replica: 26264b2caebf4120abcceb6211ef6530, State: Running, Role: LEADER
I20260924 04:03:12.690606 2711 consensus_queue.cc:237] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } }
I20260924 04:03:12.692173 2132 catalog_manager.cc:5495] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26264b2caebf4120abcceb6211ef6530 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "26264b2caebf4120abcceb6211ef6530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26264b2caebf4120abcceb6211ef6530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 } health_report { overall_health: UNKNOWN } } }
I20260924 04:03:12.724160 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:12.721252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.0.98.2:35825)
0924 04:03:12.721367 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:12.721371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:12.721383 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:03:12.721554 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:12.721558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:12.721606 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:12.721665 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:12.721891 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.722400 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.723192 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:12.723406 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:12.723491 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:12.723716 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:12.723785 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:12.723840 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:12.723975 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 04:03:12.724025 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:12.729478 2288 raft_consensus.cc:1261] T 3b30fdb78b6947b29c6c0ada9703a6b3 P b186b40397ec41faaa08dd73495aecbb [term 1 FOLLOWER]: Refusing update from remote peer 26264b2caebf4120abcceb6211ef6530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:12.729461 2569 raft_consensus.cc:1261] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Refusing update from remote peer 26264b2caebf4120abcceb6211ef6530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:12.729839 2711 consensus_queue.cc:1035] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34688313822c433db985899599b188d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:12.730026 2714 consensus_queue.cc:1035] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b186b40397ec41faaa08dd73495aecbb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:03:12.745786 2701 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:03:14.298960 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:14.299113 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:14.301730 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 694912fc2e4f4ffaa9c8fbe75230595f: Sending DeleteTablet for 3 replicas of tablet 3b30fdb78b6947b29c6c0ada9703a6b3
I20260924 04:03:14.302309 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b30fdb78b6947b29c6c0ada9703a6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34322
I20260924 04:03:14.302328 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b30fdb78b6947b29c6c0ada9703a6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260924 04:03:14.302311 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b30fdb78b6947b29c6c0ada9703a6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45164
I20260924 04:03:14.302707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 2204
I20260924 04:03:14.303656 2786 tablet_replica.cc:330] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: stopping tablet replica
I20260924 04:03:14.303834 2786 raft_consensus.cc:2229] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:14.304059 2786 raft_consensus.cc:2258] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:14.304184 2785 tablet_replica.cc:330] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: stopping tablet replica
I20260924 04:03:14.304304 2785 raft_consensus.cc:2229] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:14.304411 2785 raft_consensus.cc:2258] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:14.304689 2786 ts_tablet_manager.cc:1905] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:14.305377 2785 ts_tablet_manager.cc:1905] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:14.307371 2786 ts_tablet_manager.cc:1918] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:03:14.307447 2786 log.cc:1196] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/3b30fdb78b6947b29c6c0ada9703a6b3
I20260924 04:03:14.307560 2785 ts_tablet_manager.cc:1918] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:03:14.307641 2785 log.cc:1196] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/3b30fdb78b6947b29c6c0ada9703a6b3
I20260924 04:03:14.307694 2786 ts_tablet_manager.cc:1939] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 26264b2caebf4120abcceb6211ef6530: Deleting consensus metadata
I20260924 04:03:14.307930 2785 ts_tablet_manager.cc:1939] T 3b30fdb78b6947b29c6c0ada9703a6b3 P 34688313822c433db985899599b188d7: Deleting consensus metadata
I20260924 04:03:14.308426 2117 catalog_manager.cc:4841] TS 26264b2caebf4120abcceb6211ef6530 (127.0.98.2:35825): tablet 3b30fdb78b6947b29c6c0ada9703a6b3 (table test-table [id=c11534a7046d44ed8c5811c0c5593ee6]) successfully deleted
I20260924 04:03:14.308568 2119 catalog_manager.cc:4841] TS 34688313822c433db985899599b188d7 (127.0.98.3:37231): tablet 3b30fdb78b6947b29c6c0ada9703a6b3 (table test-table [id=c11534a7046d44ed8c5811c0c5593ee6]) successfully deleted
W20260924 04:03:14.310400 2117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33265 (error 108)
I20260924 04:03:14.311126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 2344
I20260924 04:03:14.311434 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:14.311004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33265 (local address 127.0.0.1:34344)
0924 04:03:14.311254 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:14.311332 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
W20260924 04:03:14.313031 2117 catalog_manager.cc:4584] TS b186b40397ec41faaa08dd73495aecbb (127.0.98.1:33265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b30fdb78b6947b29c6c0ada9703a6b3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33265: connect: Connection refused (error 111)
I20260924 04:03:14.317509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 2484
I20260924 04:03:14.322851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 2100
2026-09-24T04:03:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6314 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:14 dist-test-slave-zcsk krb5kdc[2792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:14 dist-test-slave-zcsk krb5kdc[2792](info): set up 2 sockets
Sep 24 04:03:14 dist-test-slave-zcsk krb5kdc[2792](info): commencing operation
krb5kdc: starting...
W20260924 04:03:15.279950 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:15 dist-test-slave-zcsk krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:15.404328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34907
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:15.512403 2805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:15.512703 2805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:15.512745 2805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:15.516762 2805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:15.516819 2805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:15.516839 2805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:15.516852 2805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:15.516870 2805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:15.522169 2805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:15.523195 2805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:15.524214 2805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:15.532667 2813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:15.532720 2810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:15.532685 2811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:15.532773 2805 server_base.cc:1034] running on GCE node
I20260924 04:03:15.533505 2805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:15.534572 2805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:15.535816 2805 hybrid_clock.cc:648] HybridClock initialized: now 1790222595535798 us; error 30 us; skew 500 ppm
Sep 24 04:03:15 dist-test-slave-zcsk krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:15.538489 2805 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:15.539302 2805 webserver.cc:460] Webserver started at http://127.0.98.62:43101/ using document root <none> and password file <none>
I20260924 04:03:15.539826 2805 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:15.539881 2805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:15.540089 2805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:15.542066 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "10100a31ef3d4c69b10bb366fc632245"
format_stamp: "Formatted at 2026-09-24 04:03:15 on dist-test-slave-zcsk"
server_key: "b62c124c7c2b14693182d938ca6ada72"
server_key_iv: "52c15fb6b8e342bcf19512b4e9090673"
server_key_version: "encryptionkey@0"
I20260924 04:03:15.542603 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "10100a31ef3d4c69b10bb366fc632245"
format_stamp: "Formatted at 2026-09-24 04:03:15 on dist-test-slave-zcsk"
server_key: "b62c124c7c2b14693182d938ca6ada72"
server_key_iv: "52c15fb6b8e342bcf19512b4e9090673"
server_key_version: "encryptionkey@0"
I20260924 04:03:15.546568 2805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:03:15.548971 2820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:15.550292 2805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:15.550410 2805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "10100a31ef3d4c69b10bb366fc632245"
format_stamp: "Formatted at 2026-09-24 04:03:15 on dist-test-slave-zcsk"
server_key: "b62c124c7c2b14693182d938ca6ada72"
server_key_iv: "52c15fb6b8e342bcf19512b4e9090673"
server_key_version: "encryptionkey@0"
I20260924 04:03:15.550508 2805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:15.568249 2805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:15.571370 2805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:15.571556 2805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:15.578296 2805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34907
I20260924 04:03:15.578315 2872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34907 every 8 connection(s)
I20260924 04:03:15.579308 2805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:15.580471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 2805
I20260924 04:03:15.580567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:15.580839 392 external_mini_cluster.cc:1428] Setting key 9c06386656013e431ba8f312e040f058
I20260924 04:03:15.582429 2873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:15.588845 2873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245: Bootstrap starting.
Sep 24 04:03:15 dist-test-slave-zcsk krb5kdc[2792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:15.591133 2873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:15.591833 2873 log.cc:824] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:15.593809 2873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245: No bootstrap required, opened a new log
I20260924 04:03:15.594515 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:15.582119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.0.98.62:34907)
0924 04:03:15.582514 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:15.582532 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:15.582578 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:03:15.583239 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:15.583256 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:15.583607 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:15.583892 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:15.584802 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:15.585789 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:15.586946 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:15.587406 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:15.587518 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:15.590193 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:15.590198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:15.590212 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:15.590241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:15.591831 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:15.592274 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:15.592279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:15.592284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:15.592396 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:15.592645 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:15.592648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:15.592650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:15.593039 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:15.593230 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:15.593490 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:03:15.593767 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260924 04:03:15.597671 2873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10100a31ef3d4c69b10bb366fc632245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34907 } }
I20260924 04:03:15.597913 2873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:15.597970 2873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10100a31ef3d4c69b10bb366fc632245, State: Initialized, Role: FOLLOWER
I20260924 04:03:15.598383 2873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10100a31ef3d4c69b10bb366fc632245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34907 } }
I20260924 04:03:15.598511 2873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:15.598574 2873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:15.598671 2873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:15.599581 2873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10100a31ef3d4c69b10bb366fc632245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34907 } }
I20260924 04:03:15.599898 2873 leader_election.cc:302] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10100a31ef3d4c69b10bb366fc632245; no voters:
I20260924 04:03:15.600276 2873 leader_election.cc:288] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:15.600399 2878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:15.600662 2878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [term 1 LEADER]: Becoming Leader. State: Replica: 10100a31ef3d4c69b10bb366fc632245, State: Running, Role: LEADER
I20260924 04:03:15.600983 2878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10100a31ef3d4c69b10bb366fc632245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34907 } }
I20260924 04:03:15.601572 2873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:15.602645 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10100a31ef3d4c69b10bb366fc632245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10100a31ef3d4c69b10bb366fc632245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34907 } } }
I20260924 04:03:15.602669 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10100a31ef3d4c69b10bb366fc632245. Latest consensus state: current_term: 1 leader_uuid: "10100a31ef3d4c69b10bb366fc632245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10100a31ef3d4c69b10bb366fc632245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34907 } } }
I20260924 04:03:15.602782 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:15.602782 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:15.603121 2887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:15.606034 2887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:15.611971 2887 catalog_manager.cc:1347] Generated new cluster ID: 6cab59b73ea0402d81b438bed06cea6b
I20260924 04:03:15.612051 2887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:15.618018 2887 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:15.618968 2887 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:15.624588 2887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10100a31ef3d4c69b10bb366fc632245: Generated new TSK 0
I20260924 04:03:15.625283 2887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:15.659642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34907
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:15.783327 2899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:15.783631 2899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:15.783674 2899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:15.787869 2899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:15.787997 2899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:15.788064 2899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:15.793529 2899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:15.794649 2899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:15.795674 2899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:15.805222 2907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:15.805243 2904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:15.805270 2905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:15.805519 2899 server_base.cc:1034] running on GCE node
I20260924 04:03:15.806007 2899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:15.806625 2899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:15.807793 2899 hybrid_clock.cc:648] HybridClock initialized: now 1790222595807777 us; error 27 us; skew 500 ppm
Sep 24 04:03:15 dist-test-slave-zcsk krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:15.810573 2899 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:15.811434 2899 webserver.cc:460] Webserver started at http://127.0.98.1:45943/ using document root <none> and password file <none>
I20260924 04:03:15.812013 2899 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:15.812080 2899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:15.812274 2899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:15.814271 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "4a6073e281ed49048904be62f6fd67a0"
format_stamp: "Formatted at 2026-09-24 04:03:15 on dist-test-slave-zcsk"
server_key: "4708a4a7e5734cf68b031cca816635c3"
server_key_iv: "a9ab14bbff3d5a2777ebb394921ca985"
server_key_version: "encryptionkey@0"
I20260924 04:03:15.814812 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4a6073e281ed49048904be62f6fd67a0"
format_stamp: "Formatted at 2026-09-24 04:03:15 on dist-test-slave-zcsk"
server_key: "4708a4a7e5734cf68b031cca816635c3"
server_key_iv: "a9ab14bbff3d5a2777ebb394921ca985"
server_key_version: "encryptionkey@0"
I20260924 04:03:15.818475 2899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:03:15.820952 2914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:15.822249 2899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:15.822361 2899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "4a6073e281ed49048904be62f6fd67a0"
format_stamp: "Formatted at 2026-09-24 04:03:15 on dist-test-slave-zcsk"
server_key: "4708a4a7e5734cf68b031cca816635c3"
server_key_iv: "a9ab14bbff3d5a2777ebb394921ca985"
server_key_version: "encryptionkey@0"
I20260924 04:03:15.822487 2899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:15.837571 2899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:15.840726 2899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:15.840893 2899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:15.841457 2899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:15.842265 2899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:15.842312 2899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:15.842351 2899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:15.842372 2899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:15.850454 2899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35261
I20260924 04:03:15.850520 3027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35261 every 8 connection(s)
I20260924 04:03:15.851366 2899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:15.856658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 2899
I20260924 04:03:15.856794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:15.857074 392 external_mini_cluster.cc:1428] Setting key 6d228e8dcf5966dca12936e0ab4c1fe9
Sep 24 04:03:15 dist-test-slave-zcsk krb5kdc[2792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:03:15.866895 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:15.853253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40399 (local address 127.0.98.62:34907)
0924 04:03:15.853395 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:15.853399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:15.854347 (+ 948us) server_negotiation.cc:407] Connection header received
0924 04:03:15.855773 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:15.855776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:15.855832 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:15.855931 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:15.857485 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:15.858348 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:15.859863 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:15.860248 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:15.860333 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:15.863361 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:15.863365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:15.863367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:15.863391 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:15.864799 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:15.865332 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:15.865336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:15.865338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:15.865422 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:15.865747 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:15.865750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:15.865752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:15.865966 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:15.866062 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:15.866582 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 04:03:15.866714 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:03:15.867725 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:15.853735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34907 (local address 127.0.98.1:40399)
0924 04:03:15.854199 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:15.854244 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:15.855548 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:15.856134 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:15.856141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:15.856606 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:15.857292 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:15.857314 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:15.858493 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:15.858497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:15.859697 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:15.859710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:15.860571 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:15.860575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:15.860791 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:15.861543 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:15.861561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:15.863183 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:15.864916 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:15.864923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:15.864940 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:15.865222 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:15.865508 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:15.865511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:15.865514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:15.865658 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:15.866070 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:15.866077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:15.866335 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 04:03:15.866529 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 04:03:15.866867 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:03:15.868978 3028 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34907
I20260924 04:03:15.869216 3028 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:15.869678 3028 heartbeater.cc:511] Master 127.0.98.62:34907 requested a full tablet report, sending...
I20260924 04:03:15.871162 2837 ts_manager.cc:194] Registered new tserver with Master: 4a6073e281ed49048904be62f6fd67a0 (127.0.98.1:35261)
I20260924 04:03:15.872347 2837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:03:15.882974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34907
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:15.989753 3033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:15.990046 3033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:15.990109 3033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:15.994310 3033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:15.994388 3033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:15.994474 3033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:15.999879 3033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:16.000969 3033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:16.002067 3033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:16.011556 3038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:16.011574 3041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:16.011662 3033 server_base.cc:1034] running on GCE node
W20260924 04:03:16.011722 3039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:16.012193 3033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:16.012933 3033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:16.014092 3033 hybrid_clock.cc:648] HybridClock initialized: now 1790222596014063 us; error 51 us; skew 500 ppm
Sep 24 04:03:16 dist-test-slave-zcsk krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:16.017452 3033 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:16.018586 3033 webserver.cc:460] Webserver started at http://127.0.98.2:37995/ using document root <none> and password file <none>
I20260924 04:03:16.019313 3033 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:16.019402 3033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:16.019658 3033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:16.022027 3033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "bab56d15ea14407aab3de0f6feff62ae"
format_stamp: "Formatted at 2026-09-24 04:03:16 on dist-test-slave-zcsk"
server_key: "a7b4a48dcf54825b80af9e625809c701"
server_key_iv: "966057464a471396209e73b3b33a00a4"
server_key_version: "encryptionkey@0"
I20260924 04:03:16.022572 3033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bab56d15ea14407aab3de0f6feff62ae"
format_stamp: "Formatted at 2026-09-24 04:03:16 on dist-test-slave-zcsk"
server_key: "a7b4a48dcf54825b80af9e625809c701"
server_key_iv: "966057464a471396209e73b3b33a00a4"
server_key_version: "encryptionkey@0"
I20260924 04:03:16.026311 3033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:03:16.028851 3048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:16.030046 3033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:03:16.030177 3033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "bab56d15ea14407aab3de0f6feff62ae"
format_stamp: "Formatted at 2026-09-24 04:03:16 on dist-test-slave-zcsk"
server_key: "a7b4a48dcf54825b80af9e625809c701"
server_key_iv: "966057464a471396209e73b3b33a00a4"
server_key_version: "encryptionkey@0"
I20260924 04:03:16.030269 3033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:16.066689 3033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:16.069700 3033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:16.069876 3033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:16.070463 3033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:16.071286 3033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:16.071334 3033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:16.071388 3033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:16.071412 3033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:16.079805 3033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45213
I20260924 04:03:16.079841 3161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45213 every 8 connection(s)
I20260924 04:03:16.080684 3033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:16.089145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 3033
I20260924 04:03:16.089247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:16.089514 392 external_mini_cluster.cc:1428] Setting key 8d9e8ea7e57ea871aa85b4487223ed2b
Sep 24 04:03:16 dist-test-slave-zcsk krb5kdc[2792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:16.094566 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:16.082380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43695 (local address 127.0.98.62:34907)
0924 04:03:16.082544 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:16.082548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:16.083453 (+ 905us) server_negotiation.cc:407] Connection header received
0924 04:03:16.084850 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:16.084854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:16.084918 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:16.085015 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:16.086211 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:16.086763 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:16.087959 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:16.088189 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:16.088288 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:16.090924 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:16.090926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:16.090928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:16.090953 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:16.092473 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:16.093061 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:16.093065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:16.093067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:16.093142 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:16.093477 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:16.093479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:16.093481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:16.093666 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:16.093751 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:16.094339 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 04:03:16.094438 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 04:03:16.095777 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:16.082769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34907 (local address 127.0.98.2:43695)
0924 04:03:16.083247 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:16.083291 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:16.084633 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:16.085196 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:16.085202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:16.085565 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:16.086065 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:16.086079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:16.086897 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:16.086900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:16.087806 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:16.087816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:16.088340 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:16.088344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:16.088517 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:16.089129 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:16.089145 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:16.090748 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:16.092597 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:16.092603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:16.092620 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:16.092969 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:16.093221 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:16.093223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:16.093225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:16.093396 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:16.093760 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:16.093767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:16.094071 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 04:03:16.094294 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 04:03:16.094629 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260924 04:03:16.096951 3162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34907
I20260924 04:03:16.097216 3162 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:16.097783 3162 heartbeater.cc:511] Master 127.0.98.62:34907 requested a full tablet report, sending...
I20260924 04:03:16.098881 2837 ts_manager.cc:194] Registered new tserver with Master: bab56d15ea14407aab3de0f6feff62ae (127.0.98.2:45213)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:03:16.099433 2837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:03:16.114830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34907
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:16.222604 3167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:16.222894 3167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:16.222940 3167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:16.227056 3167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:16.227139 3167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:16.227208 3167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:16.232683 3167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:16.233784 3167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:16.234825 3167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:16.244611 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:16.244589 3175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:16.244635 3167 server_base.cc:1034] running on GCE node
W20260924 04:03:16.244596 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:16.245337 3167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:16.246019 3167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:16.247174 3167 hybrid_clock.cc:648] HybridClock initialized: now 1790222596247154 us; error 36 us; skew 500 ppm
Sep 24 04:03:16 dist-test-slave-zcsk krb5kdc[2792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:16.249878 3167 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:16.250756 3167 webserver.cc:460] Webserver started at http://127.0.98.3:42061/ using document root <none> and password file <none>
I20260924 04:03:16.251293 3167 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:16.251355 3167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:16.251554 3167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:16.253748 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "185e29b288f949b5920112255d546e18"
format_stamp: "Formatted at 2026-09-24 04:03:16 on dist-test-slave-zcsk"
server_key: "afd8205d480a0e1d8b9034e161d2b373"
server_key_iv: "57812635cc092d88e46d35c18f2cd1df"
server_key_version: "encryptionkey@0"
I20260924 04:03:16.254328 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "185e29b288f949b5920112255d546e18"
format_stamp: "Formatted at 2026-09-24 04:03:16 on dist-test-slave-zcsk"
server_key: "afd8205d480a0e1d8b9034e161d2b373"
server_key_iv: "57812635cc092d88e46d35c18f2cd1df"
server_key_version: "encryptionkey@0"
I20260924 04:03:16.258133 3167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 04:03:16.260613 3182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:16.261829 3167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:03:16.261936 3167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "185e29b288f949b5920112255d546e18"
format_stamp: "Formatted at 2026-09-24 04:03:16 on dist-test-slave-zcsk"
server_key: "afd8205d480a0e1d8b9034e161d2b373"
server_key_iv: "57812635cc092d88e46d35c18f2cd1df"
server_key_version: "encryptionkey@0"
I20260924 04:03:16.262032 3167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:16.295526 3167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:16.298502 3167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:16.298684 3167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:16.299263 3167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:16.300102 3167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:16.300149 3167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:16.300202 3167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:16.300227 3167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:16.308768 3167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35349
I20260924 04:03:16.308791 3295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35349 every 8 connection(s)
I20260924 04:03:16.309639 3167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:16.310679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 3167
I20260924 04:03:16.310782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:16.311064 392 external_mini_cluster.cc:1428] Setting key 85f20a7762202437a1ba1ecb4bf89959
Sep 24 04:03:16 dist-test-slave-zcsk krb5kdc[2792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:16.322768 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:16.311494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48165 (local address 127.0.98.62:34907)
0924 04:03:16.311613 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:16.311618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:16.312639 (+ 1021us) server_negotiation.cc:407] Connection header received
0924 04:03:16.314004 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:16.314007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:16.314059 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:16.314133 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:16.315259 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:16.315778 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:16.316932 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:16.317158 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:16.317243 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:16.319502 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:16.319505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:16.319507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:16.319533 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:16.320842 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:16.321327 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:16.321330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:16.321331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:16.321406 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:16.321722 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:16.321725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:16.321727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:16.321900 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:16.321984 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:16.322473 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 04:03:16.322605 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:03:16.323553 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:16.312016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34907 (local address 127.0.98.3:48165)
0924 04:03:16.312508 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:16.312548 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:16.313812 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:16.314303 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:16.314310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:16.314662 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:16.315130 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:16.315143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:16.315893 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:16.315896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:16.316820 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:16.316829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:16.317286 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:16.317290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:16.317458 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:16.317967 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:16.317983 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:16.319347 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:16.320946 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:16.320952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:16.320968 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:16.321234 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:16.321485 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:16.321488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:16.321490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:16.321638 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:16.321998 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:16.322006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:16.322253 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 04:03:16.322417 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:03:16.322747 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260924 04:03:16.324664 3296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34907
I20260924 04:03:16.324910 3296 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:16.325382 3296 heartbeater.cc:511] Master 127.0.98.62:34907 requested a full tablet report, sending...
I20260924 04:03:16.326359 2837 ts_manager.cc:194] Registered new tserver with Master: 185e29b288f949b5920112255d546e18 (127.0.98.3:35349)
I20260924 04:03:16.326906 2837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48165
I20260924 04:03:16.336388 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:03:16.347733 2876 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:16.346462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.0.98.62:34907)
0924 04:03:16.346595 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:16.346599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:16.346738 (+ 139us) server_negotiation.cc:407] Connection header received
0924 04:03:16.346846 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:16.346848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:16.346890 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:16.346962 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:16.347626 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44716: BlockingRecv error: recv got EOF from 127.0.0.1:44716 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:03:16.348817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 2899
I20260924 04:03:16.353457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 3033
I20260924 04:03:16.358085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 3167
I20260924 04:03:16.362560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 2805
2026-09-24T04:03:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2038 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:16 dist-test-slave-zcsk krb5kdc[3308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:16 dist-test-slave-zcsk krb5kdc[3308](info): set up 2 sockets
Sep 24 04:03:16 dist-test-slave-zcsk krb5kdc[3308](info): commencing operation
krb5kdc: starting...
W20260924 04:03:17.323632 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:17 dist-test-slave-zcsk krb5kdc[3308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:17.443432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40299
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:17.551710 3321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:17.552029 3321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:17.552071 3321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:17.556154 3321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:17.556221 3321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:17.556241 3321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:17.556257 3321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:17.556270 3321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:17.561620 3321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:17.562673 3321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:17.563647 3321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:17.572182 3326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:17.572190 3329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:17.572225 3327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:17.572227 3321 server_base.cc:1034] running on GCE node
I20260924 04:03:17.573048 3321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:17.574222 3321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:17.575378 3321 hybrid_clock.cc:648] HybridClock initialized: now 1790222597575353 us; error 43 us; skew 500 ppm
Sep 24 04:03:17 dist-test-slave-zcsk krb5kdc[3308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:17.578049 3321 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:17.578979 3321 webserver.cc:460] Webserver started at http://127.0.98.62:39025/ using document root <none> and password file <none>
I20260924 04:03:17.579490 3321 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:17.579546 3321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:17.579715 3321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:17.581805 3321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "d54cc3394dd24f0fbaf472b3155569d9"
format_stamp: "Formatted at 2026-09-24 04:03:17 on dist-test-slave-zcsk"
server_key: "fc5004947c54904f9572dbb474bc8147"
server_key_iv: "376c883c00de17b80439ecddeb74c8ac"
server_key_version: "encryptionkey@0"
I20260924 04:03:17.582374 3321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "d54cc3394dd24f0fbaf472b3155569d9"
format_stamp: "Formatted at 2026-09-24 04:03:17 on dist-test-slave-zcsk"
server_key: "fc5004947c54904f9572dbb474bc8147"
server_key_iv: "376c883c00de17b80439ecddeb74c8ac"
server_key_version: "encryptionkey@0"
I20260924 04:03:17.585985 3321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:03:17.588440 3336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:17.589751 3321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:17.589855 3321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "d54cc3394dd24f0fbaf472b3155569d9"
format_stamp: "Formatted at 2026-09-24 04:03:17 on dist-test-slave-zcsk"
server_key: "fc5004947c54904f9572dbb474bc8147"
server_key_iv: "376c883c00de17b80439ecddeb74c8ac"
server_key_version: "encryptionkey@0"
I20260924 04:03:17.589960 3321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:17.602916 3321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:17.606029 3321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:17.606189 3321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:17.613052 3321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40299
I20260924 04:03:17.613412 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40299 every 8 connection(s)
I20260924 04:03:17.614034 3321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:17.616899 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:17.619544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 3321
I20260924 04:03:17.619647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:17.619931 392 external_mini_cluster.cc:1428] Setting key d67a2ebe567eba65bf58f19e5e96ab6d
I20260924 04:03:17.622884 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9: Bootstrap starting.
I20260924 04:03:17.625447 3389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:17.626349 3389 log.cc:824] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:17.628356 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9: No bootstrap required, opened a new log
Sep 24 04:03:17 dist-test-slave-zcsk krb5kdc[3308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:17.632197 3389 raft_consensus.cc:348] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54cc3394dd24f0fbaf472b3155569d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } }
I20260924 04:03:17.632483 3389 raft_consensus.cc:374] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:17.632547 3389 raft_consensus.cc:729] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54cc3394dd24f0fbaf472b3155569d9, State: Initialized, Role: FOLLOWER
I20260924 04:03:17.633085 3389 consensus_queue.cc:260] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54cc3394dd24f0fbaf472b3155569d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } }
I20260924 04:03:17.633246 3389 raft_consensus.cc:388] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:17.633317 3389 raft_consensus.cc:482] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:17.633420 3389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:17.634572 3389 raft_consensus.cc:504] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54cc3394dd24f0fbaf472b3155569d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } }
I20260924 04:03:17.634943 3389 leader_election.cc:302] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d54cc3394dd24f0fbaf472b3155569d9; no voters:
I20260924 04:03:17.635113 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:17.621437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.0.98.62:40299)
0924 04:03:17.621901 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:17.621918 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:17.621955 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:03:17.622569 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:17.622586 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:17.622940 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:17.623245 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:17.624136 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:17.625149 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:17.626313 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:17.626809 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:17.626930 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:17.629311 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:17.629317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:17.629333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:17.629365 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:17.631883 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:17.632352 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:17.632357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:17.632363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:17.632512 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:17.632788 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:17.632791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:17.632793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:17.633314 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:17.633527 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:17.633781 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:03:17.634093 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
I20260924 04:03:17.635298 3389 leader_election.cc:288] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:17.635661 3395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:17.635856 3395 raft_consensus.cc:686] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [term 1 LEADER]: Becoming Leader. State: Replica: d54cc3394dd24f0fbaf472b3155569d9, State: Running, Role: LEADER
I20260924 04:03:17.636246 3395 consensus_queue.cc:237] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54cc3394dd24f0fbaf472b3155569d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } }
I20260924 04:03:17.636652 3389 sys_catalog.cc:564] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:17.637885 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d54cc3394dd24f0fbaf472b3155569d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54cc3394dd24f0fbaf472b3155569d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } } }
I20260924 04:03:17.638010 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:17.638019 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:03:17.638486 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d54cc3394dd24f0fbaf472b3155569d9. Latest consensus state: current_term: 1 leader_uuid: "d54cc3394dd24f0fbaf472b3155569d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54cc3394dd24f0fbaf472b3155569d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } } }
I20260924 04:03:17.638823 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9 [sys.catalog]: This master's current role is: LEADER
W20260924 04:03:17.641634 3409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:03:17.641702 3409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:03:17.641775 3404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:17.642477 3404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:17.648118 3404 catalog_manager.cc:1347] Generated new cluster ID: 6e342d2e48d64d69aca3efac2a28b338
I20260924 04:03:17.648186 3404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:17.657886 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:17.652085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.0.98.62:40299)
0924 04:03:17.652201 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:17.652206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:17.652516 (+ 310us) server_negotiation.cc:407] Connection header received
0924 04:03:17.652631 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:17.652635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:17.652687 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:17.652830 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:17.653401 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:17.654197 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:17.655328 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:17.655719 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:17.655818 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:17.656763 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:17.656766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:17.656769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:17.657394 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:17.657430 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:17.657450 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:17.657538 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:17.657665 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:03:17.657726 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:03:17.659225 3404 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:17.660279 3404 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:17.670679 3404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d54cc3394dd24f0fbaf472b3155569d9: Generated new TSK 0
I20260924 04:03:17.671635 3404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:03:17.677057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 3321
2026-09-24T04:03:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1313 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:17 dist-test-slave-zcsk krb5kdc[3422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:17 dist-test-slave-zcsk krb5kdc[3422](info): set up 2 sockets
Sep 24 04:03:17 dist-test-slave-zcsk krb5kdc[3422](info): commencing operation
krb5kdc: starting...
W20260924 04:03:18.635732 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:18 dist-test-slave-zcsk krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:18.755852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41247
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:18.864825 3435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:18.865114 3435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:18.865163 3435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:18.869277 3435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:18.869349 3435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:18.869369 3435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:18.869383 3435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:18.869398 3435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:18.874825 3435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:18.875882 3435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:18.876907 3435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:18.885182 3441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:18.885179 3440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:18.885231 3443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:18.885605 3435 server_base.cc:1034] running on GCE node
I20260924 04:03:18.886235 3435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:18.887439 3435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:18.888615 3435 hybrid_clock.cc:648] HybridClock initialized: now 1790222598888580 us; error 43 us; skew 500 ppm
Sep 24 04:03:18 dist-test-slave-zcsk krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:18.891515 3435 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:18.892508 3435 webserver.cc:460] Webserver started at http://127.0.98.62:39711/ using document root <none> and password file <none>
I20260924 04:03:18.893041 3435 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:18.893096 3435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:18.893275 3435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:18.895323 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "dab4c666c5274e72abff9ebdf4ae70c1"
format_stamp: "Formatted at 2026-09-24 04:03:18 on dist-test-slave-zcsk"
server_key: "a19549a8984654045df3939015acf44f"
server_key_iv: "bcb315539a01ee43871a9a2e770b4c8f"
server_key_version: "encryptionkey@0"
I20260924 04:03:18.895845 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "dab4c666c5274e72abff9ebdf4ae70c1"
format_stamp: "Formatted at 2026-09-24 04:03:18 on dist-test-slave-zcsk"
server_key: "a19549a8984654045df3939015acf44f"
server_key_iv: "bcb315539a01ee43871a9a2e770b4c8f"
server_key_version: "encryptionkey@0"
I20260924 04:03:18.899519 3435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:18.902055 3450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:18.903525 3435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:18.903659 3435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "dab4c666c5274e72abff9ebdf4ae70c1"
format_stamp: "Formatted at 2026-09-24 04:03:18 on dist-test-slave-zcsk"
server_key: "a19549a8984654045df3939015acf44f"
server_key_iv: "bcb315539a01ee43871a9a2e770b4c8f"
server_key_version: "encryptionkey@0"
I20260924 04:03:18.903776 3435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:18.916678 3435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:18.919657 3435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:18.919848 3435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:18.926800 3435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41247
I20260924 04:03:18.927055 3502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41247 every 8 connection(s)
I20260924 04:03:18.927814 3435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:18.930589 3503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:18.931967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 3435
I20260924 04:03:18.932073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:18.932333 392 external_mini_cluster.cc:1428] Setting key 8bbf6382b26c7e2e77d9b9ba3f86de65
I20260924 04:03:18.936534 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1: Bootstrap starting.
I20260924 04:03:18.938679 3503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:18.939456 3503 log.cc:824] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:03:18 dist-test-slave-zcsk krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:18.941488 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1: No bootstrap required, opened a new log
I20260924 04:03:18.945107 3503 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4c666c5274e72abff9ebdf4ae70c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } }
I20260924 04:03:18.945358 3503 raft_consensus.cc:374] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:18.945393 3503 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab4c666c5274e72abff9ebdf4ae70c1, State: Initialized, Role: FOLLOWER
I20260924 04:03:18.945752 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:18.933718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.0.98.62:41247)
0924 04:03:18.934166 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:18.934183 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:18.934219 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:03:18.934880 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:18.934897 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:18.935237 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:18.935545 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:18.936367 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:18.937370 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:18.938206 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:18.938670 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:18.938808 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:18.941316 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:18.941322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:18.941335 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:18.941362 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:18.942954 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:18.943364 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:18.943369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:18.943374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:18.943483 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:18.943737 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:18.943740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:18.943742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:18.944161 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:18.944348 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:18.944630 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 04:03:18.944926 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":135,"threads_started":1}
I20260924 04:03:18.945825 3503 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4c666c5274e72abff9ebdf4ae70c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } }
I20260924 04:03:18.946034 3503 raft_consensus.cc:388] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:18.946092 3503 raft_consensus.cc:482] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:18.946188 3503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:18.947069 3503 raft_consensus.cc:504] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4c666c5274e72abff9ebdf4ae70c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } }
I20260924 04:03:18.947400 3503 leader_election.cc:302] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dab4c666c5274e72abff9ebdf4ae70c1; no voters:
I20260924 04:03:18.947687 3503 leader_election.cc:288] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:18.947774 3508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:18.948025 3508 raft_consensus.cc:686] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [term 1 LEADER]: Becoming Leader. State: Replica: dab4c666c5274e72abff9ebdf4ae70c1, State: Running, Role: LEADER
I20260924 04:03:18.948314 3508 consensus_queue.cc:237] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4c666c5274e72abff9ebdf4ae70c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } }
I20260924 04:03:18.949036 3503 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:18.949985 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dab4c666c5274e72abff9ebdf4ae70c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4c666c5274e72abff9ebdf4ae70c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } }
I20260924 04:03:18.950109 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:18.950457 3517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:18.950416 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab4c666c5274e72abff9ebdf4ae70c1. Latest consensus state: current_term: 1 leader_uuid: "dab4c666c5274e72abff9ebdf4ae70c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4c666c5274e72abff9ebdf4ae70c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } }
I20260924 04:03:18.950577 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:18.953899 3517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:18.959560 3517 catalog_manager.cc:1347] Generated new cluster ID: e42632b113c14c49ae2938a87606a49e
I20260924 04:03:18.959628 3517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:18.970131 3517 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:18.971144 3517 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:18.977582 3517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dab4c666c5274e72abff9ebdf4ae70c1: Generated new TSK 0
I20260924 04:03:18.978344 3517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:19.016882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:19.128719 3529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:19.129001 3529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:19.129045 3529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:19.133033 3529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:19.133095 3529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:19.133153 3529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:19.138377 3529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:19.139379 3529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:19.140357 3529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:19.149842 3534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:19.149880 3529 server_base.cc:1034] running on GCE node
W20260924 04:03:19.149863 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:19.149842 3537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:19.150544 3529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:19.151229 3529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:19.152410 3529 hybrid_clock.cc:648] HybridClock initialized: now 1790222599152378 us; error 44 us; skew 500 ppm
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:19.155267 3529 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:19.156255 3529 webserver.cc:460] Webserver started at http://127.0.98.1:44347/ using document root <none> and password file <none>
I20260924 04:03:19.156803 3529 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:19.156864 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:19.157078 3529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:19.159140 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "72da29c199074c8a84440ea3465dcd7b"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "52dab04ac916685b6f8bcef2bdf3f5cd"
server_key_iv: "652eb8517fce8264dac46f22aa779a0c"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.159667 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72da29c199074c8a84440ea3465dcd7b"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "52dab04ac916685b6f8bcef2bdf3f5cd"
server_key_iv: "652eb8517fce8264dac46f22aa779a0c"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.163364 3529 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:03:19.165802 3544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.167024 3529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:19.167176 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "72da29c199074c8a84440ea3465dcd7b"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "52dab04ac916685b6f8bcef2bdf3f5cd"
server_key_iv: "652eb8517fce8264dac46f22aa779a0c"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.167277 3529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:19.178371 3529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:19.181144 3529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:19.181315 3529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:19.181864 3529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:19.182688 3529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:19.182736 3529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.182790 3529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:19.182817 3529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.191167 3529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36027
I20260924 04:03:19.191180 3657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36027 every 8 connection(s)
I20260924 04:03:19.192068 3529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:19.192708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 3529
I20260924 04:03:19.192818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:19.193085 392 external_mini_cluster.cc:1428] Setting key 78f09a60e33c427145a1e4d897d9dfe7
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:19.205667 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:19.193840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58493 (local address 127.0.98.62:41247)
0924 04:03:19.193972 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:19.193976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:19.194922 (+ 946us) server_negotiation.cc:407] Connection header received
0924 04:03:19.196416 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:19.196419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:19.196464 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:19.196532 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:19.197841 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.198381 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.199488 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.199720 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.199809 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:19.202183 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:19.202186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:19.202189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:19.202216 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:19.203597 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:19.204148 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:19.204151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:19.204153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:19.204233 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:19.204555 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:19.204558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:19.204560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:19.204747 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:19.204835 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:19.205358 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 04:03:19.205494 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:03:19.206483 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:19.194337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.98.1:58493)
0924 04:03:19.194787 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:19.194830 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:19.196190 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:19.196716 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:19.196725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:19.197206 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:19.197703 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:19.197716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.198496 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.198499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:19.199374 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:19.199383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.199861 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.199864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:19.200061 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:19.200590 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:19.200606 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:19.202008 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:19.203708 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:19.203715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:19.203734 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:19.204053 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:19.204315 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:19.204318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:19.204320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:19.204462 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:19.204848 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:19.204856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:19.205117 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 04:03:19.205281 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:03:19.205624 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260924 04:03:19.207732 3658 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41247
I20260924 04:03:19.208033 3658 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:19.208648 3658 heartbeater.cc:511] Master 127.0.98.62:41247 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:03:19.210176 3467 ts_manager.cc:194] Registered new tserver with Master: 72da29c199074c8a84440ea3465dcd7b (127.0.98.1:36027)
I20260924 04:03:19.211664 3467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:03:19.227511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:19.339210 3663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:19.339485 3663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:19.339529 3663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:19.343596 3663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:19.343674 3663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:19.343753 3663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:19.349100 3663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:19.350224 3663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:19.351280 3663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:19.360694 3669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:19.360699 3668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:19.361058 3663 server_base.cc:1034] running on GCE node
W20260924 04:03:19.361380 3671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:19.361804 3663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:19.362424 3663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:19.363581 3663 hybrid_clock.cc:648] HybridClock initialized: now 1790222599363559 us; error 34 us; skew 500 ppm
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:19.366258 3663 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:19.367175 3663 webserver.cc:460] Webserver started at http://127.0.98.2:43877/ using document root <none> and password file <none>
I20260924 04:03:19.367723 3663 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:19.367789 3663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:19.368011 3663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:19.370030 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "9afdf74b824941fa8ea89237781ec303"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "4257b0ce9fd3583b2e41467b0b8743a2"
server_key_iv: "ab311627b9d23ea3010363316d6d674d"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.370563 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9afdf74b824941fa8ea89237781ec303"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "4257b0ce9fd3583b2e41467b0b8743a2"
server_key_iv: "ab311627b9d23ea3010363316d6d674d"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.374284 3663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:03:19.376690 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.377897 3663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:19.378011 3663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "9afdf74b824941fa8ea89237781ec303"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "4257b0ce9fd3583b2e41467b0b8743a2"
server_key_iv: "ab311627b9d23ea3010363316d6d674d"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.378108 3663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:19.406164 3663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:19.409425 3663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:19.409603 3663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:19.410185 3663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:19.411098 3663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:19.411146 3663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.411202 3663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:19.411226 3663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.419610 3663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46575
I20260924 04:03:19.419637 3791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46575 every 8 connection(s)
I20260924 04:03:19.420532 3663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:19.423767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 3663
I20260924 04:03:19.423877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:19.424145 392 external_mini_cluster.cc:1428] Setting key 687d9ae4b5f97211046b6c5121ad6988
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:03:19.435799 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:19.422369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35365 (local address 127.0.98.62:41247)
0924 04:03:19.422489 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:19.422493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:19.423384 (+ 891us) server_negotiation.cc:407] Connection header received
0924 04:03:19.424920 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:19.424924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:19.424975 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:19.425049 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:19.426343 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.427089 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.428762 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.428985 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.429075 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:19.432069 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:19.432072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:19.432074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:19.432100 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:19.433554 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:19.434219 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:19.434223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:19.434225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:19.434309 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:19.434676 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:19.434679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:19.434680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:19.434870 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:19.434947 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:19.435523 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 04:03:19.435670 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:03:19.436761 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:19.422753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.98.2:35365)
0924 04:03:19.423228 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:19.423273 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:19.424720 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:19.425207 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:19.425215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:19.425682 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:19.426197 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:19.426212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.427226 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.427230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:19.428617 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:19.428629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.429140 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.429145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:19.429350 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:19.430002 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:19.430037 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:19.431851 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:19.433673 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:19.433680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:19.433698 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:19.434067 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:19.434400 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:19.434402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:19.434404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:19.434557 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:19.434980 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:19.434985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:19.435235 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 04:03:19.435426 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 04:03:19.435756 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260924 04:03:19.438702 3792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41247
I20260924 04:03:19.438958 3792 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:19.439455 3792 heartbeater.cc:511] Master 127.0.98.62:41247 requested a full tablet report, sending...
I20260924 04:03:19.440557 3467 ts_manager.cc:194] Registered new tserver with Master: 9afdf74b824941fa8ea89237781ec303 (127.0.98.2:46575)
I20260924 04:03:19.441145 3467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:03:19.451592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:19.560650 3797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:19.560984 3797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:19.561033 3797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:19.565111 3797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:19.565200 3797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:19.565264 3797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:19.570779 3797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:19.571971 3797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:19.573047 3797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:19.582851 3803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:19.582903 3797 server_base.cc:1034] running on GCE node
W20260924 04:03:19.582924 3802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:19.582870 3805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:19.583601 3797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:19.584313 3797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:19.585479 3797 hybrid_clock.cc:648] HybridClock initialized: now 1790222599585453 us; error 35 us; skew 500 ppm
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:19.588204 3797 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:19.589103 3797 webserver.cc:460] Webserver started at http://127.0.98.3:45623/ using document root <none> and password file <none>
I20260924 04:03:19.589640 3797 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:19.589706 3797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:19.589921 3797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:19.592108 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "71c54da34281496384346ef00be45a94"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "852874923986df0425f870706dd624d5"
server_key_iv: "25b4bf428a1c20dcb5180e7dc4635aa9"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.592659 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71c54da34281496384346ef00be45a94"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "852874923986df0425f870706dd624d5"
server_key_iv: "25b4bf428a1c20dcb5180e7dc4635aa9"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.596387 3797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:19.598775 3812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.600158 3797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:19.600276 3797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "71c54da34281496384346ef00be45a94"
format_stamp: "Formatted at 2026-09-24 04:03:19 on dist-test-slave-zcsk"
server_key: "852874923986df0425f870706dd624d5"
server_key_iv: "25b4bf428a1c20dcb5180e7dc4635aa9"
server_key_version: "encryptionkey@0"
I20260924 04:03:19.600380 3797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:19.616011 3797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:19.619407 3797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:19.619619 3797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:19.620289 3797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:19.621212 3797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:19.621268 3797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.621325 3797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:19.621358 3797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:19.630517 3797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35549
I20260924 04:03:19.630530 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35549 every 8 connection(s)
I20260924 04:03:19.631481 3797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:19.637740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 3797
I20260924 04:03:19.637871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:19.638140 392 external_mini_cluster.cc:1428] Setting key af025eb813acf52e0fd25a5a47fc0eff
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:19.645572 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:19.633444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45327 (local address 127.0.98.62:41247)
0924 04:03:19.633602 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:19.633607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:19.634581 (+ 974us) server_negotiation.cc:407] Connection header received
0924 04:03:19.636057 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:19.636060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:19.636117 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:19.636197 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:19.637494 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.638270 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.639612 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.639938 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.640055 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:19.642434 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:19.642437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:19.642439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:19.642462 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:19.643601 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:19.644125 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:19.644128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:19.644130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:19.644213 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:19.644533 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:19.644536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:19.644537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:19.644740 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:19.644825 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:19.645311 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 04:03:19.645441 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:03:19.646360 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:19.633893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.98.3:45327)
0924 04:03:19.634389 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:19.634438 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:19.635775 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:19.636377 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:19.636384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:19.636773 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:19.637329 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:19.637344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.638396 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.638398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:19.639514 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:19.639524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.640097 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.640102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:19.640294 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:19.640882 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:19.640899 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:19.642282 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:19.643694 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:19.643699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:19.643712 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:19.644018 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:19.644293 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:19.644295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:19.644296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:19.644426 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:19.644831 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:19.644836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:19.645071 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 04:03:19.645227 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 04:03:19.645488 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":108,"threads_started":1}
I20260924 04:03:19.647578 3926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41247
I20260924 04:03:19.647828 3926 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:19.648412 3926 heartbeater.cc:511] Master 127.0.98.62:41247 requested a full tablet report, sending...
I20260924 04:03:19.649588 3467 ts_manager.cc:194] Registered new tserver with Master: 71c54da34281496384346ef00be45a94 (127.0.98.3:35549)
I20260924 04:03:19.650168 3467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45327
I20260924 04:03:19.652110 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222599, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222599, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:19.677201 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:19.668692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.0.98.62:41247)
0924 04:03:19.668858 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:19.668862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:19.668953 (+ 91us) server_negotiation.cc:407] Connection header received
0924 04:03:19.669144 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:19.669147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:19.669192 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:19.669265 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:19.669988 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.670510 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.671435 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:19.671659 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:19.671758 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:19.674192 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:19.674195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:19.674197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:19.674226 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:19.675514 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:19.675971 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:19.675974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:19.675976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:19.676056 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:19.676318 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:19.676321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:19.676322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:19.676586 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:19.676699 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:19.676927 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 04:03:19.677028 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 04:03:19.677588 3467 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51270
I20260924 04:03:19.678404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 3529
I20260924 04:03:19.683275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 3663
I20260924 04:03:19.687868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 3797
I20260924 04:03:19.692233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 3435
2026-09-24T04:03:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2016 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3938](info): set up 2 sockets
Sep 24 04:03:19 dist-test-slave-zcsk krb5kdc[3938](info): commencing operation
krb5kdc: starting...
W20260924 04:03:20.655954 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:20 dist-test-slave-zcsk krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:20.777679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35015
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:20.889938 3951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:20.890328 3951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:20.890400 3951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:20.894582 3951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:20.894655 3951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:20.894675 3951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:20.894690 3951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:20.894704 3951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:20.900290 3951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:20.901382 3951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:20.902379 3951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:20.911242 3956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:20.911284 3957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:20.911253 3959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:20.911518 3951 server_base.cc:1034] running on GCE node
I20260924 04:03:20.912149 3951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:20.913340 3951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:20.914516 3951 hybrid_clock.cc:648] HybridClock initialized: now 1790222600914498 us; error 32 us; skew 500 ppm
Sep 24 04:03:20 dist-test-slave-zcsk krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:20.917541 3951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:20.918586 3951 webserver.cc:460] Webserver started at http://127.0.98.62:39405/ using document root <none> and password file <none>
I20260924 04:03:20.919152 3951 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:20.919220 3951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:20.919422 3951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:20.921564 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "654f8468e1bc43ff9c39bc8ba2911098"
format_stamp: "Formatted at 2026-09-24 04:03:20 on dist-test-slave-zcsk"
server_key: "08d2b266b2caa30bfaad322187019b33"
server_key_iv: "a7fb392b98a94e84136729e726cfd7f3"
server_key_version: "encryptionkey@0"
I20260924 04:03:20.922116 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "654f8468e1bc43ff9c39bc8ba2911098"
format_stamp: "Formatted at 2026-09-24 04:03:20 on dist-test-slave-zcsk"
server_key: "08d2b266b2caa30bfaad322187019b33"
server_key_iv: "a7fb392b98a94e84136729e726cfd7f3"
server_key_version: "encryptionkey@0"
I20260924 04:03:20.925982 3951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:03:20.928613 3966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:20.929997 3951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:20.930140 3951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "654f8468e1bc43ff9c39bc8ba2911098"
format_stamp: "Formatted at 2026-09-24 04:03:20 on dist-test-slave-zcsk"
server_key: "08d2b266b2caa30bfaad322187019b33"
server_key_iv: "a7fb392b98a94e84136729e726cfd7f3"
server_key_version: "encryptionkey@0"
I20260924 04:03:20.930258 3951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:20.942180 3951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:20.945478 3951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:20.945667 3951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:20.952801 3951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35015
I20260924 04:03:20.952824 4018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35015 every 8 connection(s)
I20260924 04:03:20.953775 3951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:20.954071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 3951
I20260924 04:03:20.954172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:20.954444 392 external_mini_cluster.cc:1428] Setting key 22f8984c98e08921d087180bad2bb119
I20260924 04:03:20.956848 4019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:20.962323 4019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098: Bootstrap starting.
Sep 24 04:03:20 dist-test-slave-zcsk krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:20.964903 4019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:20.965727 4019 log.cc:824] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:20.967926 4019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098: No bootstrap required, opened a new log
I20260924 04:03:20.968240 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:20.955818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.0.98.62:35015)
0924 04:03:20.956297 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:20.956313 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:20.956349 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:03:20.957023 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:20.957043 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:20.957411 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:20.957733 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:20.958518 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:20.959518 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:20.960368 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:20.960834 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:20.960982 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:20.963381 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:20.963389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:20.963406 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:20.963437 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:20.965277 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:20.965712 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:20.965717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:20.965724 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:20.965858 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:20.966144 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:20.966147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:20.966149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:20.966552 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:20.966731 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:20.967000 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 04:03:20.967281 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":141,"threads_started":1}
I20260924 04:03:20.971743 4019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654f8468e1bc43ff9c39bc8ba2911098" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35015 } }
I20260924 04:03:20.972014 4019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:20.972061 4019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654f8468e1bc43ff9c39bc8ba2911098, State: Initialized, Role: FOLLOWER
I20260924 04:03:20.972457 4019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654f8468e1bc43ff9c39bc8ba2911098" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35015 } }
I20260924 04:03:20.972585 4019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:20.972631 4019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:20.972700 4019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:20.973575 4019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654f8468e1bc43ff9c39bc8ba2911098" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35015 } }
I20260924 04:03:20.973881 4019 leader_election.cc:302] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 654f8468e1bc43ff9c39bc8ba2911098; no voters:
I20260924 04:03:20.974179 4019 leader_election.cc:288] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:20.974326 4024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:20.974546 4024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [term 1 LEADER]: Becoming Leader. State: Replica: 654f8468e1bc43ff9c39bc8ba2911098, State: Running, Role: LEADER
I20260924 04:03:20.974928 4024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654f8468e1bc43ff9c39bc8ba2911098" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35015 } }
I20260924 04:03:20.975379 4019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:20.976504 4025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "654f8468e1bc43ff9c39bc8ba2911098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654f8468e1bc43ff9c39bc8ba2911098" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35015 } } }
I20260924 04:03:20.976621 4025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:20.976989 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 654f8468e1bc43ff9c39bc8ba2911098. Latest consensus state: current_term: 1 leader_uuid: "654f8468e1bc43ff9c39bc8ba2911098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654f8468e1bc43ff9c39bc8ba2911098" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35015 } } }
I20260924 04:03:20.977166 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098 [sys.catalog]: This master's current role is: LEADER
W20260924 04:03:20.980185 4039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:03:20.980262 4039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:03:20.980334 4033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:20.981052 4033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:20.987084 4033 catalog_manager.cc:1347] Generated new cluster ID: 755ecea0e12a4aaaaea172598289989f
I20260924 04:03:20.987179 4033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:21.006471 4033 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:21.007503 4033 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:21.015635 4033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 654f8468e1bc43ff9c39bc8ba2911098: Generated new TSK 0
I20260924 04:03:21.016422 4033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:21.050778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35015
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:21.162225 4045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:21.162546 4045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:21.162611 4045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:21.166960 4045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:21.167045 4045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:21.167132 4045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:21.172712 4045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:21.173833 4045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:21.174885 4045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:21.185010 4051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:21.185032 4050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:21.185010 4053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:21.185143 4045 server_base.cc:1034] running on GCE node
I20260924 04:03:21.185740 4045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:21.186481 4045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:21.187667 4045 hybrid_clock.cc:648] HybridClock initialized: now 1790222601187637 us; error 50 us; skew 500 ppm
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:21.190807 4045 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:21.191768 4045 webserver.cc:460] Webserver started at http://127.0.98.1:43633/ using document root <none> and password file <none>
I20260924 04:03:21.192379 4045 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:21.192451 4045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:21.192678 4045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:21.194763 4045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "29dee0e625b84f008c14657f97debbe1"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "4382d071703dbf73e87f1448cbfdd8f3"
server_key_iv: "d4d380a6edae97ec8f99026dd2fbd420"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.195312 4045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29dee0e625b84f008c14657f97debbe1"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "4382d071703dbf73e87f1448cbfdd8f3"
server_key_iv: "d4d380a6edae97ec8f99026dd2fbd420"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.199460 4045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:03:21.202129 4060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.203509 4045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:21.203627 4045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "29dee0e625b84f008c14657f97debbe1"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "4382d071703dbf73e87f1448cbfdd8f3"
server_key_iv: "d4d380a6edae97ec8f99026dd2fbd420"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.203745 4045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:21.219698 4045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:21.222808 4045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:21.222980 4045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:21.223606 4045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:21.224495 4045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:21.224541 4045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.224582 4045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:21.224610 4045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.232980 4045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35759
I20260924 04:03:21.233057 4173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35759 every 8 connection(s)
I20260924 04:03:21.233938 4045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:21.237396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 4045
I20260924 04:03:21.237493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:21.237742 392 external_mini_cluster.cc:1428] Setting key 69a8fa5b5a179559c2553e62e1d7f2d9
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:03:21.248163 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:21.235885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36719 (local address 127.0.98.62:35015)
0924 04:03:21.236052 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:21.236056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:21.236945 (+ 889us) server_negotiation.cc:407] Connection header received
0924 04:03:21.238406 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:21.238423 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:21.238483 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:21.238553 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:21.239849 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.240406 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.241570 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.241802 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.241889 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:21.244317 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:21.244321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:21.244323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:21.244348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:21.245999 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.246508 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.246512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.246513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.246608 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.246944 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.246947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.246949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.247134 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:21.247221 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:21.247813 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 04:03:21.247992 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 04:03:21.249022 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:21.236317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35015 (local address 127.0.98.1:36719)
0924 04:03:21.236795 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:21.236838 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:21.238198 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:21.238739 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:21.238746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:21.239204 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:21.239703 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:21.239717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.240534 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.240536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:21.241448 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:21.241458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.241950 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.241954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:21.242122 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:21.242695 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:21.242711 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:21.244154 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:21.246113 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:21.246120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:21.246138 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:21.246399 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:21.246695 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:21.246698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:21.246700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:21.246849 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:21.247228 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:21.247235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:21.247578 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 04:03:21.247756 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 04:03:21.248158 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260924 04:03:21.250326 4174 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35015
I20260924 04:03:21.250595 4174 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:21.251091 4174 heartbeater.cc:511] Master 127.0.98.62:35015 requested a full tablet report, sending...
I20260924 04:03:21.252796 3983 ts_manager.cc:194] Registered new tserver with Master: 29dee0e625b84f008c14657f97debbe1 (127.0.98.1:35759)
I20260924 04:03:21.254379 3983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:03:21.267133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35015
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:21.376811 4179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:21.377134 4179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:21.377185 4179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:21.381431 4179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:21.381501 4179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:21.381564 4179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:21.387010 4179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:21.388178 4179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:21.389182 4179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:21.399044 4185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:21.399044 4179 server_base.cc:1034] running on GCE node
W20260924 04:03:21.399015 4184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:21.399261 4187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:21.399753 4179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:21.400444 4179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:21.401614 4179 hybrid_clock.cc:648] HybridClock initialized: now 1790222601401593 us; error 40 us; skew 500 ppm
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:21.404486 4179 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:21.405371 4179 webserver.cc:460] Webserver started at http://127.0.98.2:36225/ using document root <none> and password file <none>
I20260924 04:03:21.405908 4179 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:21.405967 4179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:21.406132 4179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:21.408238 4179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb1fee23963543d9b3a3ffd32c00de1f"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "661cc48c5aeb72d9cea1d4ab99ae56dd"
server_key_iv: "8308129f85ec3d51ab6984ab3e280bf4"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.408761 4179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb1fee23963543d9b3a3ffd32c00de1f"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "661cc48c5aeb72d9cea1d4ab99ae56dd"
server_key_iv: "8308129f85ec3d51ab6984ab3e280bf4"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.412487 4179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:03:21.414889 4194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.416137 4179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:21.416239 4179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "fb1fee23963543d9b3a3ffd32c00de1f"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "661cc48c5aeb72d9cea1d4ab99ae56dd"
server_key_iv: "8308129f85ec3d51ab6984ab3e280bf4"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.416325 4179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:21.432780 4179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:21.435772 4179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:21.435968 4179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:21.436519 4179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:21.437273 4179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:21.437318 4179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.437357 4179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:21.437377 4179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.445796 4179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41503
I20260924 04:03:21.445848 4307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41503 every 8 connection(s)
I20260924 04:03:21.446693 4179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:21.453886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 4179
I20260924 04:03:21.454052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:21.454355 392 external_mini_cluster.cc:1428] Setting key 4c36eea670c158f3e48bfe81b3847cf7
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:21.460088 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:21.448525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57461 (local address 127.0.98.62:35015)
0924 04:03:21.448660 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:21.448664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:21.449536 (+ 872us) server_negotiation.cc:407] Connection header received
0924 04:03:21.450839 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:21.450842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:21.450891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:21.450968 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:21.452185 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.452703 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.453820 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.454139 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.454246 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:21.456612 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:21.456616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:21.456619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:21.456646 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:21.457969 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.458499 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.458501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.458502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.458567 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.458876 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.458878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.458879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.459067 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:21.459138 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:21.459782 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 04:03:21.459961 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:03:21.461218 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:21.448923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35015 (local address 127.0.98.2:57461)
0924 04:03:21.449377 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:21.449427 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:21.450645 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:21.451133 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:21.451140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:21.451515 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:21.452059 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:21.452076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.452814 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.452817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:21.453703 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:21.453712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.454277 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.454280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:21.454455 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:21.454990 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:21.455006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:21.456455 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:21.458074 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:21.458080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:21.458094 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:21.458413 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:21.458634 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:21.458636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:21.458638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:21.458801 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:21.459160 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:21.459166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:21.459502 (+ 336us) client_negotiation.cc:769] Sending connection context
0924 04:03:21.459716 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 04:03:21.460102 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260924 04:03:21.462504 4308 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35015
I20260924 04:03:21.462733 4308 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:21.463212 4308 heartbeater.cc:511] Master 127.0.98.62:35015 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:03:21.464339 3983 ts_manager.cc:194] Registered new tserver with Master: fb1fee23963543d9b3a3ffd32c00de1f (127.0.98.2:41503)
I20260924 04:03:21.465035 3983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:03:21.483103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35015
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:21.594307 4313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:21.594624 4313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:21.594686 4313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:21.598819 4313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:21.598891 4313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:21.598973 4313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:21.604350 4313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:21.605473 4313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:21.606559 4313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:21.616506 4313 server_base.cc:1034] running on GCE node
W20260924 04:03:21.616442 4319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:21.616442 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:21.616446 4321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:21.617064 4313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:21.617735 4313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:21.618918 4313 hybrid_clock.cc:648] HybridClock initialized: now 1790222601618895 us; error 39 us; skew 500 ppm
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:21.622167 4313 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:21.623102 4313 webserver.cc:460] Webserver started at http://127.0.98.3:32791/ using document root <none> and password file <none>
I20260924 04:03:21.623652 4313 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:21.623718 4313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:21.623957 4313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:21.626067 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "01198bfc61724a7da0027ff8e5a4b1c1"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "e2b91e7f92d0d0777bcddf47fffd0293"
server_key_iv: "0bfc84d724ac7dfd9114e3c9de32817c"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.626605 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "01198bfc61724a7da0027ff8e5a4b1c1"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "e2b91e7f92d0d0777bcddf47fffd0293"
server_key_iv: "0bfc84d724ac7dfd9114e3c9de32817c"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.630443 4313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:03:21.633176 4328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.634653 4313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:03:21.634776 4313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "01198bfc61724a7da0027ff8e5a4b1c1"
format_stamp: "Formatted at 2026-09-24 04:03:21 on dist-test-slave-zcsk"
server_key: "e2b91e7f92d0d0777bcddf47fffd0293"
server_key_iv: "0bfc84d724ac7dfd9114e3c9de32817c"
server_key_version: "encryptionkey@0"
I20260924 04:03:21.634910 4313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:21.655196 4313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:21.658324 4313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:21.658524 4313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:21.659127 4313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:21.660027 4313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:21.660085 4313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.660151 4313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:21.660175 4313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:21.668624 4313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33057
I20260924 04:03:21.668658 4441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33057 every 8 connection(s)
I20260924 04:03:21.669541 4313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:21.679687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 4313
I20260924 04:03:21.679793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:21.680136 392 external_mini_cluster.cc:1428] Setting key c8933455b8fafa5d51e7f56dd5d728b9
I20260924 04:03:21.683513 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:21.671500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36983 (local address 127.0.98.62:35015)
0924 04:03:21.671635 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:21.671639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:21.672481 (+ 842us) server_negotiation.cc:407] Connection header received
0924 04:03:21.673856 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:21.673860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:21.673960 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:21.674037 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:21.675282 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.675876 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.677103 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.677330 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.677417 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:21.679873 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:21.679877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:21.679880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:21.679935 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:21.681408 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.681999 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.682002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.682004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.682091 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.682491 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.682494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.682496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.682711 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:21.682787 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:21.683282 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 04:03:21.683394 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:03:21.684423 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:21.671815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35015 (local address 127.0.98.3:36983)
0924 04:03:21.672315 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:21.672361 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:21.673638 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:21.674226 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:21.674234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:21.674616 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:21.675109 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:21.675123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.676010 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.676014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:21.676994 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:21.677004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.677476 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.677479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:21.677637 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:21.678188 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:21.678203 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:21.679676 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:21.681519 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:21.681525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:21.681542 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:21.681888 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:21.682185 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:21.682188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:21.682189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:21.682385 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:21.682793 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:21.682797 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:21.683060 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 04:03:21.683224 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:03:21.683508 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":119,"threads_started":1}
I20260924 04:03:21.685652 4442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35015
I20260924 04:03:21.685900 4442 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:21.686393 4442 heartbeater.cc:511] Master 127.0.98.62:35015 requested a full tablet report, sending...
I20260924 04:03:21.687505 3983 ts_manager.cc:194] Registered new tserver with Master: 01198bfc61724a7da0027ff8e5a4b1c1 (127.0.98.3:33057)
I20260924 04:03:21.688163 3983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36983
I20260924 04:03:21.694307 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:03:21.719013 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:21.709819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.0.98.1:35759)
0924 04:03:21.710138 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:21.710155 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:21.710182 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:03:21.710239 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:21.710243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:21.710444 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:21.710576 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:21.711219 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.712176 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.713124 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.713507 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.713596 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:21.715797 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:21.715841 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:21.715856 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:21.715882 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:21.717404 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.717776 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.717782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.717783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.717901 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.718126 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.718128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.718129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.718355 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:21.718515 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:21.718654 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 04:03:21.718781 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:21.729533 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:21.721896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.0.98.62:35015)
0924 04:03:21.722035 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:21.722039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:21.722053 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:21.722183 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:21.722186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:21.722233 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:21.722295 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:21.722892 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.723415 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.724238 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:21.724463 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:21.724554 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:21.726866 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:21.726869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:21.726872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:21.726900 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:21.728228 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.728561 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.728563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.728564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.728641 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:21.728823 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:21.728825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:21.728827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:21.729035 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:21.729130 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:21.729303 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 04:03:21.729401 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:03:21.729933 3983 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48476
I20260924 04:03:21.730531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 4045
I20260924 04:03:21.735750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 4179
I20260924 04:03:21.740679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 4313
I20260924 04:03:21.745316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 3951
2026-09-24T04:03:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2052 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[4454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[4454](info): set up 2 sockets
Sep 24 04:03:21 dist-test-slave-zcsk krb5kdc[4454](info): commencing operation
krb5kdc: starting...
W20260924 04:03:22.707803 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:22 dist-test-slave-zcsk krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:22.827440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:22.956991 4467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:22.957279 4467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:22.957322 4467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:22.961381 4467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:22.961452 4467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:22.961472 4467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:22.961488 4467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:22.961508 4467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:22.966920 4467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:22.968060 4467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:22.969067 4467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:22.977293 4473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:22.977288 4472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:22.977421 4467 server_base.cc:1034] running on GCE node
W20260924 04:03:22.977559 4475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:22.978142 4467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:22.979365 4467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:22.980574 4467 hybrid_clock.cc:648] HybridClock initialized: now 1790222602980528 us; error 74 us; skew 500 ppm
Sep 24 04:03:22 dist-test-slave-zcsk krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:22.983353 4467 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:22.984162 4467 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:22.984236 4467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:22.984447 4467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:22.986510 4467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "7c89e352e9a74d009bd16a403996b28f"
format_stamp: "Formatted at 2026-09-24 04:03:22 on dist-test-slave-zcsk"
server_key: "ba3676d70f55259b335359c56a1d30af"
server_key_iv: "2b7459b1037bf797a54a2cff4d2dfa91"
server_key_version: "encryptionkey@0"
I20260924 04:03:22.987061 4467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c89e352e9a74d009bd16a403996b28f"
format_stamp: "Formatted at 2026-09-24 04:03:22 on dist-test-slave-zcsk"
server_key: "ba3676d70f55259b335359c56a1d30af"
server_key_iv: "2b7459b1037bf797a54a2cff4d2dfa91"
server_key_version: "encryptionkey@0"
I20260924 04:03:22.991065 4467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:03:22.993839 4481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:22.995255 4467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:22.995391 4467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "7c89e352e9a74d009bd16a403996b28f"
format_stamp: "Formatted at 2026-09-24 04:03:22 on dist-test-slave-zcsk"
server_key: "ba3676d70f55259b335359c56a1d30af"
server_key_iv: "2b7459b1037bf797a54a2cff4d2dfa91"
server_key_version: "encryptionkey@0"
I20260924 04:03:22.995500 4467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:23.030957 4467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:23.034235 4467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:23.034420 4467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:23.041355 4467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46111
I20260924 04:03:23.041589 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46111 every 8 connection(s)
I20260924 04:03:23.042024 4467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:23.044090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 4467
I20260924 04:03:23.044193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:23.044445 392 external_mini_cluster.cc:1428] Setting key 901c5cfd257f0fb1197973ef40371a85
I20260924 04:03:23.044925 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:23.051894 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f: Bootstrap starting.
Sep 24 04:03:23 dist-test-slave-zcsk krb5kdc[4454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:23.054236 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:23.055121 4534 log.cc:824] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:23.057370 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f: No bootstrap required, opened a new log
I20260924 04:03:23.059419 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.045838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.0.98.62:46111)
0924 04:03:23.046252 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.046272 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.046317 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:03:23.047120 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.047145 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.047593 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.048018 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:23.048659 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.050092 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.050994 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.051662 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.051821 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.054190 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:23.054197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:23.054215 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:23.054250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:23.056003 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.056465 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.056470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.056476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.056616 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.056949 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.056952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.056953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.057459 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:23.057632 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.057942 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.058235 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260924 04:03:23.061182 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c89e352e9a74d009bd16a403996b28f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } }
I20260924 04:03:23.061412 4534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:23.061456 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c89e352e9a74d009bd16a403996b28f, State: Initialized, Role: FOLLOWER
I20260924 04:03:23.061887 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c89e352e9a74d009bd16a403996b28f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } }
I20260924 04:03:23.062008 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:23.062068 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:23.062157 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:23.063014 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c89e352e9a74d009bd16a403996b28f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } }
I20260924 04:03:23.063333 4534 leader_election.cc:302] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c89e352e9a74d009bd16a403996b28f; no voters:
I20260924 04:03:23.063625 4534 leader_election.cc:288] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:23.063710 4539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:23.063941 4539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [term 1 LEADER]: Becoming Leader. State: Replica: 7c89e352e9a74d009bd16a403996b28f, State: Running, Role: LEADER
I20260924 04:03:23.064513 4539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c89e352e9a74d009bd16a403996b28f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } }
I20260924 04:03:23.064845 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:23.066017 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c89e352e9a74d009bd16a403996b28f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c89e352e9a74d009bd16a403996b28f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } } }
I20260924 04:03:23.066121 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:23.066413 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c89e352e9a74d009bd16a403996b28f. Latest consensus state: current_term: 1 leader_uuid: "7c89e352e9a74d009bd16a403996b28f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c89e352e9a74d009bd16a403996b28f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } } }
I20260924 04:03:23.066493 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f [sys.catalog]: This master's current role is: LEADER
W20260924 04:03:23.069381 4554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:03:23.069447 4554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:03:23.069528 4549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:23.070413 4549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:23.075937 4549 catalog_manager.cc:1347] Generated new cluster ID: f653105884194efdb65483b992d1a2b9
I20260924 04:03:23.076012 4549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:23.101480 4549 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:23.102512 4549 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:23.121387 4549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f: Generated new TSK 0
I20260924 04:03:23.122138 4549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:23.154515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:23.267408 4560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:23.267700 4560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:23.267758 4560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:23.272053 4560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:23.272125 4560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:23.272195 4560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:23.277777 4560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:23.278925 4560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:23.280094 4560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:23.290053 4565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:23.290091 4560 server_base.cc:1034] running on GCE node
W20260924 04:03:23.290042 4568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:23.290450 4566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:23.290915 4560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:23.291604 4560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:23.292843 4560 hybrid_clock.cc:648] HybridClock initialized: now 1790222603292800 us; error 64 us; skew 500 ppm
Sep 24 04:03:23 dist-test-slave-zcsk krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:23.296020 4560 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:23.296790 4560 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:23.296861 4560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:23.297035 4560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:23.299180 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "7cc8ccdb032a462db2ce2615192cbb0f"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "844c1426aa9fcdb04adbb222c72af900"
server_key_iv: "975fce236b868e79f99ca54ef0023f85"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.299762 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7cc8ccdb032a462db2ce2615192cbb0f"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "844c1426aa9fcdb04adbb222c72af900"
server_key_iv: "975fce236b868e79f99ca54ef0023f85"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.303756 4560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:03:23.306407 4574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.308109 4560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:23.308238 4560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "7cc8ccdb032a462db2ce2615192cbb0f"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "844c1426aa9fcdb04adbb222c72af900"
server_key_iv: "975fce236b868e79f99ca54ef0023f85"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.308375 4560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:23.333113 4560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:23.336396 4560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:23.336575 4560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:23.337177 4560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:23.337996 4560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:23.338042 4560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.338085 4560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:23.338114 4560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.346140 4560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37725
I20260924 04:03:23.346164 4687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37725 every 8 connection(s)
I20260924 04:03:23.346698 4560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:23.350963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 4560
I20260924 04:03:23.351064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:23.351312 392 external_mini_cluster.cc:1428] Setting key ae663e0c80b5e79a60f19808ed00d32a
Sep 24 04:03:23 dist-test-slave-zcsk krb5kdc[4454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:03:23.360795 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.348378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43773 (local address 127.0.98.62:46111)
0924 04:03:23.348510 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.348513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.349335 (+ 822us) server_negotiation.cc:407] Connection header received
0924 04:03:23.350735 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.350739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.350801 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.350874 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:23.352243 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.352873 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.354093 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.354329 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.354416 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.356861 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:23.356866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:23.356869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:23.356898 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:23.358509 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.359015 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.359019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.359021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.359107 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.359521 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.359525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.359527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.359767 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:23.359863 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.360507 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.360614 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:03:23.361676 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.348743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46111 (local address 127.0.98.1:43773)
0924 04:03:23.349192 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.349235 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:23.350506 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:23.351063 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:23.351073 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:23.351471 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:23.352095 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.352115 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.353017 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.353020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.353971 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.353981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.354467 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.354471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.354644 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.355200 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:23.355218 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:23.356690 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:23.358616 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:23.358621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:23.358635 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:23.358914 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:23.359239 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:23.359242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:23.359244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:23.359416 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:23.359885 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:23.359890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:23.360231 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 04:03:23.360449 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 04:03:23.360774 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:03:23.363088 4688 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46111
I20260924 04:03:23.363351 4688 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:23.363888 4688 heartbeater.cc:511] Master 127.0.98.62:46111 requested a full tablet report, sending...
I20260924 04:03:23.365532 4498 ts_manager.cc:194] Registered new tserver with Master: 7cc8ccdb032a462db2ce2615192cbb0f (127.0.98.1:37725)
I20260924 04:03:23.366753 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:03:23.378937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:23.488214 4693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:23.488481 4693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:23.488544 4693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:23.492540 4693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:23.492599 4693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:23.492681 4693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:23.497918 4693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:23.499017 4693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:23.500106 4693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:23.509104 4701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:23.509137 4693 server_base.cc:1034] running on GCE node
W20260924 04:03:23.509215 4699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:23.509248 4698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:23.509806 4693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:23.510396 4693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:23.511550 4693 hybrid_clock.cc:648] HybridClock initialized: now 1790222603511526 us; error 37 us; skew 500 ppm
Sep 24 04:03:23 dist-test-slave-zcsk krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:23.514303 4693 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:23.515002 4693 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:23.515066 4693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:23.515255 4693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:23.517280 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "11cf5f71a607421f80e4c1d0b63ad0b2"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "baea813071587629dc1547807e1d3c55"
server_key_iv: "9d6661db35d1b18245540b06a54faa0d"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.517802 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "11cf5f71a607421f80e4c1d0b63ad0b2"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "baea813071587629dc1547807e1d3c55"
server_key_iv: "9d6661db35d1b18245540b06a54faa0d"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.521329 4693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:23.523756 4707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.525108 4693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:23.525226 4693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "11cf5f71a607421f80e4c1d0b63ad0b2"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "baea813071587629dc1547807e1d3c55"
server_key_iv: "9d6661db35d1b18245540b06a54faa0d"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.525333 4693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:23.540297 4693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:23.543555 4693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:23.543756 4693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:23.544380 4693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:23.545259 4693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:23.545312 4693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.545361 4693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:23.545394 4693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.554132 4693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44341
I20260924 04:03:23.554147 4820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44341 every 8 connection(s)
I20260924 04:03:23.554716 4693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:03:23 dist-test-slave-zcsk krb5kdc[4454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:23.564929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 4693
I20260924 04:03:23.565032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:23.565374 392 external_mini_cluster.cc:1428] Setting key 90c0ab1a5b725c03f63f6daa5437167f
I20260924 04:03:23.568265 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.556424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35583 (local address 127.0.98.62:46111)
0924 04:03:23.556556 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.556560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.557411 (+ 851us) server_negotiation.cc:407] Connection header received
0924 04:03:23.558719 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.558723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.558778 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.558853 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:23.560020 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.560557 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.561671 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.561892 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.561979 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.564423 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:23.564427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:23.564429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:23.564454 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:23.565953 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.566575 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.566578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.566580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.566696 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.567064 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.567068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.567070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.567290 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:23.567387 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.567970 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.568092 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:03:23.569248 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.556838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46111 (local address 127.0.98.2:35583)
0924 04:03:23.557269 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.557310 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:23.558516 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:23.559017 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:23.559025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:23.559376 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:23.559857 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.559871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.560675 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.560678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.561554 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.561563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.562029 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.562033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.562198 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.562823 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:23.562840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:23.564232 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:23.566061 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:23.566066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:23.566081 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:23.566433 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:23.566777 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:23.566780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:23.566781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:23.566963 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:23.567398 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:23.567404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:23.567685 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 04:03:23.567881 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 04:03:23.568197 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260924 04:03:23.570381 4821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46111
I20260924 04:03:23.570640 4821 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:23.571187 4821 heartbeater.cc:511] Master 127.0.98.62:46111 requested a full tablet report, sending...
I20260924 04:03:23.572351 4498 ts_manager.cc:194] Registered new tserver with Master: 11cf5f71a607421f80e4c1d0b63ad0b2 (127.0.98.2:44341)
I20260924 04:03:23.572950 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35583
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:03:23.590721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:23.699229 4826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:23.699518 4826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:23.699581 4826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:23.703629 4826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:23.703694 4826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:23.703783 4826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:23.709051 4826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:23.710088 4826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:23.711151 4826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:23.720649 4826 server_base.cc:1034] running on GCE node
W20260924 04:03:23.720602 4834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:23.720613 4831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:23.720618 4832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:23.721230 4826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:23.721874 4826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:23.723042 4826 hybrid_clock.cc:648] HybridClock initialized: now 1790222603723015 us; error 39 us; skew 500 ppm
Sep 24 04:03:23 dist-test-slave-zcsk krb5kdc[4454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:23.725804 4826 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:23.726526 4826 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:23.726585 4826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:23.726752 4826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:23.728916 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "63da88f45cab48d1bd3b4622fd1cda41"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "1586ad3108c6625c3d22f676d78a649a"
server_key_iv: "c537db30290f7ae405faf1a50a06b2b8"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.729492 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "63da88f45cab48d1bd3b4622fd1cda41"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "1586ad3108c6625c3d22f676d78a649a"
server_key_iv: "c537db30290f7ae405faf1a50a06b2b8"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.733294 4826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:03:23.735831 4840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.737150 4826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:23.737278 4826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "63da88f45cab48d1bd3b4622fd1cda41"
format_stamp: "Formatted at 2026-09-24 04:03:23 on dist-test-slave-zcsk"
server_key: "1586ad3108c6625c3d22f676d78a649a"
server_key_iv: "c537db30290f7ae405faf1a50a06b2b8"
server_key_version: "encryptionkey@0"
I20260924 04:03:23.737401 4826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:23.750393 4826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:23.753698 4826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:23.753881 4826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:23.754468 4826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:23.755318 4826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:23.755373 4826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.755415 4826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:23.755445 4826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:23.764564 4826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40977
I20260924 04:03:23.764638 4953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40977 every 8 connection(s)
I20260924 04:03:23.765213 4826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:23.766732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 4826
I20260924 04:03:23.766882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:23.767242 392 external_mini_cluster.cc:1428] Setting key 3fac871b22ec48761708dc5cfda04eb0
Sep 24 04:03:23 dist-test-slave-zcsk krb5kdc[4454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:23.778894 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.767119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41987 (local address 127.0.98.62:46111)
0924 04:03:23.767248 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.767251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.768363 (+ 1112us) server_negotiation.cc:407] Connection header received
0924 04:03:23.769943 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.769946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.769999 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.770066 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:23.771221 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.771735 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.772865 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.773085 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.773176 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.775500 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:23.775503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:23.775506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:23.775534 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:23.776945 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.777453 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.777457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.777459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.777537 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.777849 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.777852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.777854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.778037 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:23.778127 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.778608 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.778729 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:03:23.779721 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.767664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46111 (local address 127.0.98.3:41987)
0924 04:03:23.768212 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.768270 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:23.769746 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:23.770240 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:23.770247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:23.770603 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:23.771083 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.771096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.771846 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.771850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.772752 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.772761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.773214 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.773218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.773397 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.773933 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:23.773949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:23.775339 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:23.777064 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:23.777070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:23.777086 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:23.777353 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:23.777624 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:23.777626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:23.777629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:23.777766 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:23.778129 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:23.778136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:23.778386 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 04:03:23.778557 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 04:03:23.778894 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":164,"threads_started":1}
I20260924 04:03:23.780850 4954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46111
I20260924 04:03:23.781085 4954 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:23.781559 4954 heartbeater.cc:511] Master 127.0.98.62:46111 requested a full tablet report, sending...
I20260924 04:03:23.782495 4494 ts_manager.cc:194] Registered new tserver with Master: 63da88f45cab48d1bd3b4622fd1cda41 (127.0.98.3:40977)
I20260924 04:03:23.783071 4494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41987
I20260924 04:03:23.793349 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:03:23.802508 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.795275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.0.98.62:46111)
0924 04:03:23.795381 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.795385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.795585 (+ 200us) server_negotiation.cc:407] Connection header received
0924 04:03:23.795657 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.795660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.795705 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.795781 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:23.796504 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.797025 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.797829 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.798033 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.798105 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.799347 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:23.799350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:23.799352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:23.799371 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:23.800942 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.801333 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.801336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.801337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.801406 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:23.801652 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:23.801656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:23.801657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:23.801910 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:23.802038 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.802278 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.802384 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:03:23.805553 4494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:23.808564 4494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:23.823181 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.816536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37725 (local address 127.0.0.1:42820)
0924 04:03:23.816996 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.817042 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:23.817164 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:23.818300 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:23.818305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:23.818326 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:23.818506 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.818513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.820024 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.820028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.821619 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.821629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.822538 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.822542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.822706 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.822744 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:03:23.822859 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:03:23.823004 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":213,"spinlock_wait_cycles":17152,"thread_start_us":59,"threads_started":1}
I20260924 04:03:23.823261 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.817248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.0.98.2:44341)
0924 04:03:23.817830 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.817845 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.817872 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:03:23.817947 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.817952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.818173 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.818339 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:23.818664 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.819847 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.821761 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.822539 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.822788 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.822922 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.822977 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.823077 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":122,"threads_started":1}
I20260924 04:03:23.823695 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.816717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.0.98.1:37725)
0924 04:03:23.817841 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.817848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.817871 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:03:23.817944 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.817949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.818159 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.818299 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:23.818627 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.819841 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.821757 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.822320 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.822499 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.822671 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.823420 (+ 749us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.823534 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":71,"threads_started":1}
I20260924 04:03:23.823772 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.817526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.0.98.3:40977)
0924 04:03:23.818594 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.818605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.818623 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:03:23.818680 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.818683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.818812 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.818921 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:23.819126 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.819850 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.821533 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.822435 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.822544 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.822792 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.823260 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.823617 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":176,"threads_started":1}
I20260924 04:03:23.823182 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.816738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44341 (local address 127.0.0.1:58980)
0924 04:03:23.817530 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.817543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:23.817623 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:23.818325 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:23.818329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:23.818369 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:23.818520 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.818526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.820040 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.820043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.821621 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.821629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.822793 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.822796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.822879 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.822899 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:03:23.822948 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 04:03:23.823028 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":181,"spinlock_wait_cycles":2944,"thread_start_us":103,"threads_started":1}
I20260924 04:03:23.824206 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.817309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40977 (local address 127.0.0.1:37682)
0924 04:03:23.817593 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.817604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:23.818034 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:23.818917 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:23.818920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:23.818935 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:23.819029 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.819035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.819991 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.819994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.821399 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.821409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.822873 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.822877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.822977 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.822999 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:03:23.823067 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:03:23.823134 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":302,"thread_start_us":92,"threads_started":1}
I20260924 04:03:23.826592 4755 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfbbb78cc5b4cee95716e3aefab1acc (DEFAULT_TABLE table=test-table [id=6ee1a3b5ec2b468c9394ffcf70c794f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:23.826720 4888 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfbbb78cc5b4cee95716e3aefab1acc (DEFAULT_TABLE table=test-table [id=6ee1a3b5ec2b468c9394ffcf70c794f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:23.826718 4622 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfbbb78cc5b4cee95716e3aefab1acc (DEFAULT_TABLE table=test-table [id=6ee1a3b5ec2b468c9394ffcf70c794f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:23.827584 4755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfbbb78cc5b4cee95716e3aefab1acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:23.827713 4622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfbbb78cc5b4cee95716e3aefab1acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:23.827728 4888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfbbb78cc5b4cee95716e3aefab1acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:23.833586 4971 tablet_bootstrap.cc:492] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: Bootstrap starting.
I20260924 04:03:23.835427 4973 tablet_bootstrap.cc:492] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: Bootstrap starting.
I20260924 04:03:23.836227 4972 tablet_bootstrap.cc:492] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f: Bootstrap starting.
I20260924 04:03:23.836336 4971 tablet_bootstrap.cc:654] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:23.837352 4973 tablet_bootstrap.cc:654] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:23.837658 4971 log.cc:824] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:23.838313 4972 tablet_bootstrap.cc:654] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:23.838402 4973 log.cc:824] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:23.839430 4972 log.cc:824] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:23.839890 4971 tablet_bootstrap.cc:492] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: No bootstrap required, opened a new log
I20260924 04:03:23.840080 4973 tablet_bootstrap.cc:492] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: No bootstrap required, opened a new log
I20260924 04:03:23.840178 4971 ts_tablet_manager.cc:1397] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260924 04:03:23.840339 4973 ts_tablet_manager.cc:1397] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:03:23.842068 4972 tablet_bootstrap.cc:492] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f: No bootstrap required, opened a new log
I20260924 04:03:23.842315 4972 ts_tablet_manager.cc:1397] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:03:23.843477 4971 raft_consensus.cc:348] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.843731 4971 raft_consensus.cc:374] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:23.843804 4971 raft_consensus.cc:729] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11cf5f71a607421f80e4c1d0b63ad0b2, State: Initialized, Role: FOLLOWER
I20260924 04:03:23.844146 4973 raft_consensus.cc:348] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.844295 4971 consensus_queue.cc:260] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.844417 4973 raft_consensus.cc:374] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:23.844494 4973 raft_consensus.cc:729] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63da88f45cab48d1bd3b4622fd1cda41, State: Initialized, Role: FOLLOWER
I20260924 04:03:23.844926 4973 consensus_queue.cc:260] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.845211 4971 ts_tablet_manager.cc:1428] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260924 04:03:23.845573 4821 heartbeater.cc:503] Master 127.0.98.62:46111 was elected leader, sending a full tablet report...
I20260924 04:03:23.845574 4954 heartbeater.cc:503] Master 127.0.98.62:46111 was elected leader, sending a full tablet report...
I20260924 04:03:23.845886 4973 ts_tablet_manager.cc:1428] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:03:23.846014 4972 raft_consensus.cc:348] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.846282 4972 raft_consensus.cc:374] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:23.846336 4972 raft_consensus.cc:729] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc8ccdb032a462db2ce2615192cbb0f, State: Initialized, Role: FOLLOWER
I20260924 04:03:23.847262 4972 consensus_queue.cc:260] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.847978 4688 heartbeater.cc:503] Master 127.0.98.62:46111 was elected leader, sending a full tablet report...
I20260924 04:03:23.848254 4972 ts_tablet_manager.cc:1428] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:03:23.875497 4979 raft_consensus.cc:482] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:23.875710 4979 raft_consensus.cc:504] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.876892 4979 leader_election.cc:288] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11cf5f71a607421f80e4c1d0b63ad0b2 (127.0.98.2:44341), 63da88f45cab48d1bd3b4622fd1cda41 (127.0.98.3:40977)
I20260924 04:03:23.880579 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.877247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47663 (local address 127.0.98.3:40977)
0924 04:03:23.877362 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.877366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.877559 (+ 193us) server_negotiation.cc:407] Connection header received
0924 04:03:23.877672 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.877674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.877718 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.877790 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:23.878015 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.878515 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.879628 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.880041 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.880127 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.880191 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.880379 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.880451 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:03:23.880589 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.877214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46401 (local address 127.0.98.2:44341)
0924 04:03:23.877338 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.877343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.877519 (+ 176us) server_negotiation.cc:407] Connection header received
0924 04:03:23.877617 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.877621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.877673 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.877727 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:23.878017 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.878516 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.879645 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.880089 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.880189 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.880257 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.880418 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.880489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:03:23.880599 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.877136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40977 (local address 127.0.98.1:47663)
0924 04:03:23.877464 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.877476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:23.877581 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:23.877809 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:23.877812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:23.877826 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:23.877920 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.877925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.878613 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.878616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.879512 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.879521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.880145 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.880149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.880251 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.880275 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:03:23.880352 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:03:23.880425 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":26,"thread_start_us":135,"threads_started":1}
I20260924 04:03:23.880609 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.877081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44341 (local address 127.0.98.1:46401)
0924 04:03:23.877423 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.877435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:23.877529 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:23.877755 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:23.877758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:23.877774 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:23.877903 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.877909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.878626 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.878630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.879526 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:23.879534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.880197 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.880200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:23.880294 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.880325 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:03:23.880396 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:03:23.880465 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":32,"thread_start_us":105,"threads_started":1}
I20260924 04:03:23.881340 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfbbb78cc5b4cee95716e3aefab1acc" candidate_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63da88f45cab48d1bd3b4622fd1cda41" is_pre_election: true
I20260924 04:03:23.881434 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfbbb78cc5b4cee95716e3aefab1acc" candidate_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" is_pre_election: true
I20260924 04:03:23.881644 4908 raft_consensus.cc:2454] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc8ccdb032a462db2ce2615192cbb0f in term 0.
I20260924 04:03:23.881702 4775 raft_consensus.cc:2454] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc8ccdb032a462db2ce2615192cbb0f in term 0.
I20260924 04:03:23.882112 4576 leader_election.cc:302] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63da88f45cab48d1bd3b4622fd1cda41, 7cc8ccdb032a462db2ce2615192cbb0f; no voters:
I20260924 04:03:23.882383 4979 raft_consensus.cc:2781] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:23.882463 4979 raft_consensus.cc:482] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:23.882494 4979 raft_consensus.cc:3037] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:23.883474 4979 raft_consensus.cc:504] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.883826 4979 leader_election.cc:288] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [CANDIDATE]: Term 1 election: Requested vote from peers 11cf5f71a607421f80e4c1d0b63ad0b2 (127.0.98.2:44341), 63da88f45cab48d1bd3b4622fd1cda41 (127.0.98.3:40977)
I20260924 04:03:23.884207 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfbbb78cc5b4cee95716e3aefab1acc" candidate_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63da88f45cab48d1bd3b4622fd1cda41"
I20260924 04:03:23.884207 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfbbb78cc5b4cee95716e3aefab1acc" candidate_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2"
I20260924 04:03:23.884320 4908 raft_consensus.cc:3037] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:23.884330 4775 raft_consensus.cc:3037] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:23.885476 4775 raft_consensus.cc:2454] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc8ccdb032a462db2ce2615192cbb0f in term 1.
I20260924 04:03:23.885474 4908 raft_consensus.cc:2454] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc8ccdb032a462db2ce2615192cbb0f in term 1.
I20260924 04:03:23.885751 4576 leader_election.cc:302] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63da88f45cab48d1bd3b4622fd1cda41, 7cc8ccdb032a462db2ce2615192cbb0f; no voters:
I20260924 04:03:23.885916 4979 raft_consensus.cc:2781] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:23.886135 4979 raft_consensus.cc:686] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [term 1 LEADER]: Becoming Leader. State: Replica: 7cc8ccdb032a462db2ce2615192cbb0f, State: Running, Role: LEADER
I20260924 04:03:23.886492 4979 consensus_queue.cc:237] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } }
I20260924 04:03:23.889792 4496 catalog_manager.cc:5495] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cc8ccdb032a462db2ce2615192cbb0f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc8ccdb032a462db2ce2615192cbb0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 } health_report { overall_health: UNKNOWN } } }
I20260924 04:03:23.909714 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.906292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.0.98.1:37725)
0924 04:03:23.906412 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:23.906417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:23.906432 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:03:23.906543 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:23.906545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:23.906640 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:23.906734 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:23.906966 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.907457 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.908330 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:23.908544 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:23.908637 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:23.908912 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:23.909341 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:23.909446 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:23.909510 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 04:03:23.909569 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:03:23.915488 4775 raft_consensus.cc:1261] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 1 FOLLOWER]: Refusing update from remote peer 7cc8ccdb032a462db2ce2615192cbb0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:23.915488 4908 raft_consensus.cc:1261] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 1 FOLLOWER]: Refusing update from remote peer 7cc8ccdb032a462db2ce2615192cbb0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:23.916203 4982 consensus_queue.cc:1035] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63da88f45cab48d1bd3b4622fd1cda41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:23.916410 4979 consensus_queue.cc:1035] T 5bfbbb78cc5b4cee95716e3aefab1acc P 7cc8ccdb032a462db2ce2615192cbb0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "11cf5f71a607421f80e4c1d0b63ad0b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:23.924209 4985 mvcc.cc:205] Tried to move back new op lower bound from 7332751785631047680 to 7332751785521651712. Current Snapshot: MvccSnapshot[applied={T|T < 7332751785631047680 or (T in {7332751785631047680})}]
I20260924 04:03:23.924319 4986 mvcc.cc:205] Tried to move back new op lower bound from 7332751785631047680 to 7332751785521651712. Current Snapshot: MvccSnapshot[applied={T|T < 7332751785631047680}]
I20260924 04:03:23.926756 4987 mvcc.cc:205] Tried to move back new op lower bound from 7332751785631047680 to 7332751785521651712. Current Snapshot: MvccSnapshot[applied={T|T < 7332751785631047680}]
I20260924 04:03:23.930739 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:23.930892 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:23.933123 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c89e352e9a74d009bd16a403996b28f: Sending DeleteTablet for 3 replicas of tablet 5bfbbb78cc5b4cee95716e3aefab1acc
I20260924 04:03:23.933902 4622 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfbbb78cc5b4cee95716e3aefab1acc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42820
I20260924 04:03:23.934090 4755 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfbbb78cc5b4cee95716e3aefab1acc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58980
I20260924 04:03:23.934111 4888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfbbb78cc5b4cee95716e3aefab1acc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37682
I20260924 04:03:23.934634 4995 tablet_replica.cc:330] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: stopping tablet replica
I20260924 04:03:23.934705 4996 tablet_replica.cc:330] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: stopping tablet replica
I20260924 04:03:23.935012 4995 raft_consensus.cc:2229] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:23.935039 4996 raft_consensus.cc:2229] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:23.935274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 4560
I20260924 04:03:23.936340 4995 raft_consensus.cc:2258] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:23.936920 4996 raft_consensus.cc:2258] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:23.937844 4995 ts_tablet_manager.cc:1905] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:23.938280 4996 ts_tablet_manager.cc:1905] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:23.940673 4995 ts_tablet_manager.cc:1918] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:23.940850 4995 log.cc:1196] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/5bfbbb78cc5b4cee95716e3aefab1acc
I20260924 04:03:23.940995 4996 ts_tablet_manager.cc:1918] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:23.941169 4996 log.cc:1196] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/5bfbbb78cc5b4cee95716e3aefab1acc
W20260924 04:03:23.941141 4482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37725 (error 108)
I20260924 04:03:23.941256 4995 ts_tablet_manager.cc:1939] T 5bfbbb78cc5b4cee95716e3aefab1acc P 63da88f45cab48d1bd3b4622fd1cda41: Deleting consensus metadata
I20260924 04:03:23.941360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 4693
I20260924 04:03:23.941545 4996 ts_tablet_manager.cc:1939] T 5bfbbb78cc5b4cee95716e3aefab1acc P 11cf5f71a607421f80e4c1d0b63ad0b2: Deleting consensus metadata
I20260924 04:03:23.942555 4483 catalog_manager.cc:4841] TS 63da88f45cab48d1bd3b4622fd1cda41 (127.0.98.3:40977): tablet 5bfbbb78cc5b4cee95716e3aefab1acc (table test-table [id=6ee1a3b5ec2b468c9394ffcf70c794f3]) successfully deleted
I20260924 04:03:23.942957 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.942436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37725 (local address 127.0.0.1:42836)
0924 04:03:23.942820 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.942878 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311}
W20260924 04:03:23.943094 4482 catalog_manager.cc:4584] TS 7cc8ccdb032a462db2ce2615192cbb0f (127.0.98.1:37725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bfbbb78cc5b4cee95716e3aefab1acc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37725: connect: Connection refused (error 111)
I20260924 04:03:23.947309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 4826
I20260924 04:03:23.948652 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:23.947540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44341 (local address 127.0.0.1:58992)
0924 04:03:23.948527 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:23.948575 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":872}
I20260924 04:03:23.951941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 4467
2026-09-24T04:03:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2207 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T04:03:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:23Z Disabled control of system clock
I20260924 04:03:23.979717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38153
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:03:24.090252 5001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:24.090538 5001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:24.090629 5001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:24.094641 5001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:24.094697 5001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:24.094710 5001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:24.094726 5001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:24.094740 5001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:24.099416 5001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:24.100433 5001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:24.101764 5001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:24.110073 5009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:24.110028 5006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:24.110133 5007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:24.110189 5001 server_base.cc:1034] running on GCE node
I20260924 04:03:24.110852 5001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:24.111981 5001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:24.113155 5001 hybrid_clock.cc:648] HybridClock initialized: now 1790222604113130 us; error 44 us; skew 500 ppm
I20260924 04:03:24.115068 5001 webserver.cc:460] Webserver started at http://127.0.98.62:41115/ using document root <none> and password file <none>
I20260924 04:03:24.115688 5001 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:24.115764 5001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:24.116021 5001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:24.118029 5001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "154f540562c840cb802b7f5a0cc2d4c6"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2549dda15249e180f645584ffa294268"
server_key_iv: "d5f79788be83cf9a624223cb2ddf1070"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.118571 5001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "154f540562c840cb802b7f5a0cc2d4c6"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2549dda15249e180f645584ffa294268"
server_key_iv: "d5f79788be83cf9a624223cb2ddf1070"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.122176 5001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:03:24.124606 5015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.125860 5001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:24.125972 5001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "154f540562c840cb802b7f5a0cc2d4c6"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2549dda15249e180f645584ffa294268"
server_key_iv: "d5f79788be83cf9a624223cb2ddf1070"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.126062 5001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:24.130441 5001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:24.131021 5001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:24.131171 5001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:24.137502 5001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38153
I20260924 04:03:24.137501 5067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38153 every 8 connection(s)
I20260924 04:03:24.138415 5001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:24.141530 5068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:24.145015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5001
I20260924 04:03:24.145134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:24.145370 392 external_mini_cluster.cc:1428] Setting key 0f63f78b7863cbaadc6f7265d0036842
I20260924 04:03:24.147425 5068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6: Bootstrap starting.
I20260924 04:03:24.150173 5068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:24.151027 5068 log.cc:824] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:24.151252 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.146658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.0.98.62:38153)
0924 04:03:24.147327 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.147346 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.147389 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:03:24.148222 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.148245 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.148697 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.149070 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.149736 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.149741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.149756 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.149766 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.149831 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.149852 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.149956 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.150086 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.150342 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260924 04:03:24.153141 5068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6: No bootstrap required, opened a new log
I20260924 04:03:24.156889 5068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f540562c840cb802b7f5a0cc2d4c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38153 } }
I20260924 04:03:24.157122 5068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:24.157155 5068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154f540562c840cb802b7f5a0cc2d4c6, State: Initialized, Role: FOLLOWER
I20260924 04:03:24.157507 5068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f540562c840cb802b7f5a0cc2d4c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38153 } }
I20260924 04:03:24.157614 5068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:24.157675 5068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:24.157768 5068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:24.158758 5068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f540562c840cb802b7f5a0cc2d4c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38153 } }
I20260924 04:03:24.159044 5068 leader_election.cc:302] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 154f540562c840cb802b7f5a0cc2d4c6; no voters:
I20260924 04:03:24.159317 5068 leader_election.cc:288] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:24.159427 5073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:24.159638 5073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [term 1 LEADER]: Becoming Leader. State: Replica: 154f540562c840cb802b7f5a0cc2d4c6, State: Running, Role: LEADER
I20260924 04:03:24.159981 5073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f540562c840cb802b7f5a0cc2d4c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38153 } }
I20260924 04:03:24.160476 5068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:24.161459 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "154f540562c840cb802b7f5a0cc2d4c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f540562c840cb802b7f5a0cc2d4c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38153 } } }
I20260924 04:03:24.161592 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:24.161895 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 154f540562c840cb802b7f5a0cc2d4c6. Latest consensus state: current_term: 1 leader_uuid: "154f540562c840cb802b7f5a0cc2d4c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f540562c840cb802b7f5a0cc2d4c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38153 } } }
I20260924 04:03:24.161973 5082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:24.161980 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:24.164592 5082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:24.170102 5082 catalog_manager.cc:1347] Generated new cluster ID: 04008f624bb84a43be1c4e1d711e26fb
I20260924 04:03:24.170166 5082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:24.180313 5082 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:24.180625 5082 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:24.188947 5082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6: Generated new TSK 0
I20260924 04:03:24.189807 5082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:03:24.199990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38153
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:03:24.311444 5092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:24.311787 5092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:24.311854 5092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:24.316386 5092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:24.316440 5092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:24.316499 5092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:24.321226 5092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:24.322279 5092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:24.323642 5092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:24.333272 5098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:24.333293 5097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:24.333284 5092 server_base.cc:1034] running on GCE node
W20260924 04:03:24.333314 5100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:24.333915 5092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:24.334537 5092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:24.335690 5092 hybrid_clock.cc:648] HybridClock initialized: now 1790222604335671 us; error 33 us; skew 500 ppm
I20260924 04:03:24.337527 5092 webserver.cc:460] Webserver started at http://127.0.98.1:34617/ using document root <none> and password file <none>
I20260924 04:03:24.338070 5092 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:24.338129 5092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:24.338328 5092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:24.340603 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "55a4d588d16f40f5b04a53e9c9262672"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2b21f37dd12301a3610bb3ecf738907f"
server_key_iv: "a119711969d045bfbf8b78a97a4f7af1"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.341116 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "55a4d588d16f40f5b04a53e9c9262672"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2b21f37dd12301a3610bb3ecf738907f"
server_key_iv: "a119711969d045bfbf8b78a97a4f7af1"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.344539 5092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:24.346761 5106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.347882 5092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:24.348011 5092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "55a4d588d16f40f5b04a53e9c9262672"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2b21f37dd12301a3610bb3ecf738907f"
server_key_iv: "a119711969d045bfbf8b78a97a4f7af1"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.348097 5092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:24.352301 5092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:24.352854 5092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:24.353001 5092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:24.353504 5092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:24.354352 5092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:24.354398 5092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.354447 5092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:24.354476 5092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.362668 5092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42897
I20260924 04:03:24.362704 5219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42897 every 8 connection(s)
I20260924 04:03:24.363623 5092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:24.366108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5092
I20260924 04:03:24.366207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:24.366508 392 external_mini_cluster.cc:1428] Setting key 010bd957fb092b894b2199c6dd12ba55
I20260924 04:03:24.368626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38153
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:03:24.370476 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.365567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39249 (local address 127.0.98.62:38153)
0924 04:03:24.365760 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.365765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.367025 (+ 1260us) server_negotiation.cc:407] Connection header received
0924 04:03:24.368546 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.368549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.368590 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.368672 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.370003 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.370007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.370017 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.370032 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.370057 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.370068 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.370177 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.370296 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.370340 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 04:03:24.371497 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.366035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38153 (local address 127.0.98.1:39249)
0924 04:03:24.366871 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:24.366920 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:24.368310 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:24.368885 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:24.368894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:24.369018 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:24.369664 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:24.369666 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:24.369723 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:24.369816 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:24.370173 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:24.370179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:24.370183 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:03:24.370316 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 04:03:24.370527 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
I20260924 04:03:24.372712 5220 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38153
I20260924 04:03:24.372964 5220 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:24.373463 5220 heartbeater.cc:511] Master 127.0.98.62:38153 requested a full tablet report, sending...
I20260924 04:03:24.375000 5032 ts_manager.cc:194] Registered new tserver with Master: 55a4d588d16f40f5b04a53e9c9262672 (127.0.98.1:42897)
W20260924 04:03:24.479138 5223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:24.479414 5223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:24.479454 5223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:24.483505 5223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:24.483578 5223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:24.483649 5223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:24.488579 5223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:24.489722 5223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:24.491142 5223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:24.500506 5231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:24.500519 5229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:24.500522 5228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:24.500738 5223 server_base.cc:1034] running on GCE node
I20260924 04:03:24.501184 5223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:24.501792 5223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:24.502952 5223 hybrid_clock.cc:648] HybridClock initialized: now 1790222604502928 us; error 39 us; skew 500 ppm
I20260924 04:03:24.510483 5223 webserver.cc:460] Webserver started at http://127.0.98.2:40865/ using document root <none> and password file <none>
I20260924 04:03:24.511086 5223 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:24.511168 5223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:24.511377 5223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:24.513514 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "7752547d502849f5895a5bb506bb45e1"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2c82d1647e7768212cda45011f06fc4e"
server_key_iv: "ef0e2e9fafc6fe04d51b5816b571c652"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.514050 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7752547d502849f5895a5bb506bb45e1"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2c82d1647e7768212cda45011f06fc4e"
server_key_iv: "ef0e2e9fafc6fe04d51b5816b571c652"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.517813 5223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 04:03:24.520328 5237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.521641 5223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:24.521756 5223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "7752547d502849f5895a5bb506bb45e1"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "2c82d1647e7768212cda45011f06fc4e"
server_key_iv: "ef0e2e9fafc6fe04d51b5816b571c652"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.521862 5223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:24.526530 5223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:24.527114 5223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:24.527282 5223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:24.527818 5223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:24.528725 5223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:24.528774 5223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.528824 5223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:24.528851 5223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.537314 5223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35467
I20260924 04:03:24.537338 5350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35467 every 8 connection(s)
I20260924 04:03:24.538194 5223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:24.543933 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.539927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58201 (local address 127.0.98.62:38153)
0924 04:03:24.540042 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.540046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.541155 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 04:03:24.542427 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.542430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.542472 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.542546 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.543549 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.543552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.543554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.543565 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.543585 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.543593 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.543697 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.543767 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.543804 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 04:03:24.544795 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.540370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38153 (local address 127.0.98.2:58201)
0924 04:03:24.541010 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:24.541050 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:24.542215 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:24.542700 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:24.542707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:24.542798 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:24.543292 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:24.543294 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:24.543336 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:24.543407 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:24.543664 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:24.543669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:24.543672 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:03:24.543797 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:03:24.544026 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260924 04:03:24.545292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5223
I20260924 04:03:24.545400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:24.545650 392 external_mini_cluster.cc:1428] Setting key 06a8fb4e545d420b06f06f2b352cd664
I20260924 04:03:24.545867 5351 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38153
I20260924 04:03:24.546092 5351 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:24.546545 5351 heartbeater.cc:511] Master 127.0.98.62:38153 requested a full tablet report, sending...
I20260924 04:03:24.547462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38153
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:03:24.547498 5032 ts_manager.cc:194] Registered new tserver with Master: 7752547d502849f5895a5bb506bb45e1 (127.0.98.2:35467)
W20260924 04:03:24.656747 5354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:24.657025 5354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:24.657065 5354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:24.661232 5354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:24.661363 5354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:24.661460 5354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:24.666486 5354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:24.667586 5354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:24.668998 5354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:24.678982 5362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:24.679142 5354 server_base.cc:1034] running on GCE node
W20260924 04:03:24.679037 5360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:24.678965 5359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:24.679605 5354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:24.680305 5354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:24.681468 5354 hybrid_clock.cc:648] HybridClock initialized: now 1790222604681460 us; error 30 us; skew 500 ppm
I20260924 04:03:24.683372 5354 webserver.cc:460] Webserver started at http://127.0.98.3:41823/ using document root <none> and password file <none>
I20260924 04:03:24.683979 5354 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:24.684052 5354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:24.684262 5354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:24.686365 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "55ff9bd5b640445abfa03e7cff2aa356"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "9d9ad306e644fc006cadc874345e4c98"
server_key_iv: "6059d666dce3732df784b5cddbbcd699"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.686913 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "55ff9bd5b640445abfa03e7cff2aa356"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "9d9ad306e644fc006cadc874345e4c98"
server_key_iv: "6059d666dce3732df784b5cddbbcd699"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.690889 5354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:03:24.693588 5368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.695027 5354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:03:24.695156 5354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "55ff9bd5b640445abfa03e7cff2aa356"
format_stamp: "Formatted at 2026-09-24 04:03:24 on dist-test-slave-zcsk"
server_key: "9d9ad306e644fc006cadc874345e4c98"
server_key_iv: "6059d666dce3732df784b5cddbbcd699"
server_key_version: "encryptionkey@0"
I20260924 04:03:24.695278 5354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:24.700033 5354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:24.700718 5354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:24.700882 5354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:24.701435 5354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:24.702356 5354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:24.702405 5354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.702458 5354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:24.702484 5354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:24.710939 5354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40379
I20260924 04:03:24.710973 5481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40379 every 8 connection(s)
I20260924 04:03:24.711853 5354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:24.713114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5354
I20260924 04:03:24.713229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:24.713524 392 external_mini_cluster.cc:1428] Setting key b7b0f92ccc6ed62a4687e25e1e7466b2
I20260924 04:03:24.718539 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.713820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47525 (local address 127.0.98.62:38153)
0924 04:03:24.713965 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.713970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.715381 (+ 1411us) server_negotiation.cc:407] Connection header received
0924 04:03:24.716892 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.716895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.716934 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.717012 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.718187 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.718191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.718193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.718203 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.718223 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.718230 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.718299 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.718397 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.718434 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:03:24.719447 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.714338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38153 (local address 127.0.98.3:47525)
0924 04:03:24.715221 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:24.715281 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:24.716684 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:24.717218 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:24.717226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:24.717324 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:24.717923 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:24.717924 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:24.717969 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:24.718043 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:24.718294 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:24.718299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:24.718302 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:03:24.718435 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 04:03:24.718674 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":153,"threads_started":1}
I20260924 04:03:24.720566 5482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38153
I20260924 04:03:24.720780 5482 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:24.721164 5482 heartbeater.cc:511] Master 127.0.98.62:38153 requested a full tablet report, sending...
I20260924 04:03:24.722193 5032 ts_manager.cc:194] Registered new tserver with Master: 55ff9bd5b640445abfa03e7cff2aa356 (127.0.98.3:40379)
I20260924 04:03:24.727676 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:03:24.731158 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.729667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.0.98.62:38153)
0924 04:03:24.729790 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.729794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.729962 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:03:24.730082 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.730084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.730117 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.730185 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.730829 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.730833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.730836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.730848 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.730870 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.730878 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.730939 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.731011 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.731043 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:03:24.733824 5032 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:24.737378 5032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:24.749727 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.746068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42897 (local address 127.0.0.1:57132)
0924 04:03:24.746896 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:24.746950 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:24.747090 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:24.748258 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:24.748263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:24.748343 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:24.748963 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:24.748965 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:24.748972 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:24.749045 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:24.749363 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:24.749371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:24.749376 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:03:24.749480 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:03:24.749548 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":75,"threads_started":1}
I20260924 04:03:24.749992 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.746535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.0.98.3:40379)
0924 04:03:24.746842 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.746852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.747182 (+ 330us) server_negotiation.cc:407] Connection header received
0924 04:03:24.747229 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.747234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.747412 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.747524 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.749203 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.749209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.749221 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.749234 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.749280 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.749312 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.749406 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.749787 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.749848 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":128,"threads_started":1}
I20260924 04:03:24.749729 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.746196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40379 (local address 127.0.0.1:40442)
0924 04:03:24.746946 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:24.746957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:24.747060 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:24.748256 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:24.748262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:24.748344 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:24.748946 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:24.748949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:24.748970 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:24.749044 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:24.749447 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:24.749450 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:24.749453 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:03:24.749521 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:03:24.749557 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":131,"threads_started":1}
I20260924 04:03:24.750185 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.747001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.0.98.2:35467)
0924 04:03:24.747387 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.747397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.747621 (+ 224us) server_negotiation.cc:407] Connection header received
0924 04:03:24.747791 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.747797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.747981 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.748070 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.749223 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.749227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.749235 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.749246 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.749280 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.749312 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.749406 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.749559 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.749670 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":119,"threads_started":1}
I20260924 04:03:24.750211 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.746200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.0.98.1:42897)
0924 04:03:24.746657 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.746668 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.747042 (+ 374us) server_negotiation.cc:407] Connection header received
0924 04:03:24.747204 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.747210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.747412 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.747524 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.749183 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.749190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.749204 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.749217 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.749259 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.749285 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.749381 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.749493 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.749567 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":127,"threads_started":1}
I20260924 04:03:24.750587 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.746829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35467 (local address 127.0.0.1:35694)
0924 04:03:24.747521 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:24.747534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:24.747676 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:24.748539 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:24.748541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:24.748554 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:24.749081 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:24.749083 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:24.749094 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:24.749121 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:24.749447 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:24.749450 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:24.749453 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:03:24.750446 (+ 993us) client_negotiation.cc:240] Negotiation successful
0924 04:03:24.750481 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":84,"threads_started":1}
I20260924 04:03:24.753186 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7be2b3e6774983bf111d2c8daa25ed (DEFAULT_TABLE table=test-table [id=458fccfef45a4203b77fff4f3405f31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:24.753265 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7be2b3e6774983bf111d2c8daa25ed (DEFAULT_TABLE table=test-table [id=458fccfef45a4203b77fff4f3405f31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:24.753399 5285 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7be2b3e6774983bf111d2c8daa25ed (DEFAULT_TABLE table=test-table [id=458fccfef45a4203b77fff4f3405f31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:24.754156 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7be2b3e6774983bf111d2c8daa25ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:24.754261 5285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7be2b3e6774983bf111d2c8daa25ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:24.754261 5154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7be2b3e6774983bf111d2c8daa25ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:24.760471 5499 tablet_bootstrap.cc:492] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Bootstrap starting.
I20260924 04:03:24.761164 5500 tablet_bootstrap.cc:492] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672: Bootstrap starting.
I20260924 04:03:24.762042 5501 tablet_bootstrap.cc:492] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Bootstrap starting.
I20260924 04:03:24.763381 5499 tablet_bootstrap.cc:654] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:24.763381 5500 tablet_bootstrap.cc:654] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:24.764169 5501 tablet_bootstrap.cc:654] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:24.764326 5500 log.cc:824] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:24.764505 5499 log.cc:824] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:24.764953 5501 log.cc:824] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:24.766382 5500 tablet_bootstrap.cc:492] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672: No bootstrap required, opened a new log
I20260924 04:03:24.766572 5500 ts_tablet_manager.cc:1397] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:03:24.766877 5501 tablet_bootstrap.cc:492] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: No bootstrap required, opened a new log
I20260924 04:03:24.766878 5499 tablet_bootstrap.cc:492] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: No bootstrap required, opened a new log
I20260924 04:03:24.767058 5501 ts_tablet_manager.cc:1397] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:03:24.767058 5499 ts_tablet_manager.cc:1397] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:03:24.770464 5500 raft_consensus.cc:348] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.770705 5500 raft_consensus.cc:374] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:24.770756 5500 raft_consensus.cc:729] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a4d588d16f40f5b04a53e9c9262672, State: Initialized, Role: FOLLOWER
I20260924 04:03:24.771189 5500 consensus_queue.cc:260] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.771373 5501 raft_consensus.cc:348] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.771374 5499 raft_consensus.cc:348] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.771643 5501 raft_consensus.cc:374] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:24.771641 5499 raft_consensus.cc:374] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:24.771701 5501 raft_consensus.cc:729] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7752547d502849f5895a5bb506bb45e1, State: Initialized, Role: FOLLOWER
I20260924 04:03:24.771749 5499 raft_consensus.cc:729] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ff9bd5b640445abfa03e7cff2aa356, State: Initialized, Role: FOLLOWER
I20260924 04:03:24.772039 5220 heartbeater.cc:503] Master 127.0.98.62:38153 was elected leader, sending a full tablet report...
I20260924 04:03:24.772198 5499 consensus_queue.cc:260] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.772198 5501 consensus_queue.cc:260] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.772397 5500 ts_tablet_manager.cc:1428] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:03:24.773023 5351 heartbeater.cc:503] Master 127.0.98.62:38153 was elected leader, sending a full tablet report...
I20260924 04:03:24.773147 5499 ts_tablet_manager.cc:1428] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 04:03:24.773139 5501 ts_tablet_manager.cc:1428] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 04:03:24.774058 5482 heartbeater.cc:503] Master 127.0.98.62:38153 was elected leader, sending a full tablet report...
W20260924 04:03:24.789119 5352 tablet.cc:2369] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:03:24.864797 5221 tablet.cc:2369] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:24.942808 5505 raft_consensus.cc:482] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:24.943023 5505 raft_consensus.cc:504] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.944233 5505 leader_election.cc:288] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55ff9bd5b640445abfa03e7cff2aa356 (127.0.98.3:40379), 7752547d502849f5895a5bb506bb45e1 (127.0.98.2:35467)
I20260924 04:03:24.946328 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.944714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46635 (local address 127.0.98.2:35467)
0924 04:03:24.944816 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.944820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.944998 (+ 178us) server_negotiation.cc:407] Connection header received
0924 04:03:24.945111 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.945114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.945153 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.945231 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.945907 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.945911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.945914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.945927 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.945950 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.945960 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.946058 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.946143 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.946186 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:24.946329 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.944622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45465 (local address 127.0.98.3:40379)
0924 04:03:24.944763 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.944767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.945131 (+ 364us) server_negotiation.cc:407] Connection header received
0924 04:03:24.945218 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.945221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.945257 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.945324 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.945987 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.945990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.945993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.946005 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.946025 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.946034 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.946110 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.946193 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.946228 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:03:24.946360 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.944544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40379 (local address 127.0.98.1:45465)
0924 04:03:24.945046 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:24.945060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:24.945144 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:24.945353 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:24.945356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:24.945369 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:24.945838 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:24.945840 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:24.945848 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:24.945868 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:24.946102 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:24.946105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:24.946108 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:03:24.946182 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:03:24.946205 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":43,"thread_start_us":85,"threads_started":1}
I20260924 04:03:24.946360 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.944544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35467 (local address 127.0.98.1:46635)
0924 04:03:24.944908 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:24.944921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:24.945022 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:24.945277 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:24.945279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:24.945292 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:24.945750 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:24.945752 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:24.945763 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:24.945795 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:24.946044 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:24.946046 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:24.946048 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:03:24.946146 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:03:24.946185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":110,"threads_started":1}
I20260924 04:03:24.947043 5436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7be2b3e6774983bf111d2c8daa25ed" candidate_uuid: "55a4d588d16f40f5b04a53e9c9262672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ff9bd5b640445abfa03e7cff2aa356" is_pre_election: true
I20260924 04:03:24.947094 5305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7be2b3e6774983bf111d2c8daa25ed" candidate_uuid: "55a4d588d16f40f5b04a53e9c9262672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7752547d502849f5895a5bb506bb45e1" is_pre_election: true
I20260924 04:03:24.947325 5436 raft_consensus.cc:2454] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a4d588d16f40f5b04a53e9c9262672 in term 0.
I20260924 04:03:24.947348 5305 raft_consensus.cc:2454] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a4d588d16f40f5b04a53e9c9262672 in term 0.
I20260924 04:03:24.947801 5108 leader_election.cc:302] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a4d588d16f40f5b04a53e9c9262672, 7752547d502849f5895a5bb506bb45e1; no voters:
I20260924 04:03:24.948076 5505 raft_consensus.cc:2781] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:24.948164 5505 raft_consensus.cc:482] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:24.948212 5505 raft_consensus.cc:3037] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:24.949180 5505 raft_consensus.cc:504] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.949474 5505 leader_election.cc:288] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [CANDIDATE]: Term 1 election: Requested vote from peers 55ff9bd5b640445abfa03e7cff2aa356 (127.0.98.3:40379), 7752547d502849f5895a5bb506bb45e1 (127.0.98.2:35467)
I20260924 04:03:24.949817 5305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7be2b3e6774983bf111d2c8daa25ed" candidate_uuid: "55a4d588d16f40f5b04a53e9c9262672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7752547d502849f5895a5bb506bb45e1"
I20260924 04:03:24.949816 5436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7be2b3e6774983bf111d2c8daa25ed" candidate_uuid: "55a4d588d16f40f5b04a53e9c9262672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ff9bd5b640445abfa03e7cff2aa356"
I20260924 04:03:24.949924 5305 raft_consensus.cc:3037] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:24.949927 5436 raft_consensus.cc:3037] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:24.950969 5436 raft_consensus.cc:2454] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a4d588d16f40f5b04a53e9c9262672 in term 1.
I20260924 04:03:24.950974 5305 raft_consensus.cc:2454] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a4d588d16f40f5b04a53e9c9262672 in term 1.
I20260924 04:03:24.951231 5107 leader_election.cc:302] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a4d588d16f40f5b04a53e9c9262672, 55ff9bd5b640445abfa03e7cff2aa356; no voters:
I20260924 04:03:24.951435 5505 raft_consensus.cc:2781] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:24.951634 5505 raft_consensus.cc:686] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 1 LEADER]: Becoming Leader. State: Replica: 55a4d588d16f40f5b04a53e9c9262672, State: Running, Role: LEADER
I20260924 04:03:24.951992 5505 consensus_queue.cc:237] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } }
I20260924 04:03:24.955101 5031 catalog_manager.cc:5495] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a4d588d16f40f5b04a53e9c9262672 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "55a4d588d16f40f5b04a53e9c9262672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a4d588d16f40f5b04a53e9c9262672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } health_report { overall_health: UNKNOWN } } }
W20260924 04:03:24.962791 5483 tablet.cc:2369] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:24.987939 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:24.986464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.98.1:42897)
0924 04:03:24.986600 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:24.986604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:24.986619 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:03:24.986727 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:24.986729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:24.986789 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:24.986855 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:24.987548 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:24.987551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:24.987553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:24.987563 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:24.987582 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:24.987590 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:24.987663 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:24.987747 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:03:24.987785 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:03:24.994347 5436 raft_consensus.cc:1261] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 1 FOLLOWER]: Refusing update from remote peer 55a4d588d16f40f5b04a53e9c9262672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:24.994347 5305 raft_consensus.cc:1261] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 1 FOLLOWER]: Refusing update from remote peer 55a4d588d16f40f5b04a53e9c9262672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:24.995280 5505 consensus_queue.cc:1035] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7752547d502849f5895a5bb506bb45e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:24.995507 5510 consensus_queue.cc:1035] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ff9bd5b640445abfa03e7cff2aa356" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:25.003357 5513 mvcc.cc:205] Tried to move back new op lower bound from 7332751790048530432 to 7332751789886320640. Current Snapshot: MvccSnapshot[applied={T|T < 7332751790048530432}]
I20260924 04:03:25.003507 5514 mvcc.cc:205] Tried to move back new op lower bound from 7332751790048530432 to 7332751789886320640. Current Snapshot: MvccSnapshot[applied={T|T < 7332751790048530432}]
I20260924 04:03:25.011857 5027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:25.012061 5027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:25.014376 5027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 154f540562c840cb802b7f5a0cc2d4c6: Sending DeleteTablet for 3 replicas of tablet 1f7be2b3e6774983bf111d2c8daa25ed
I20260924 04:03:25.015146 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7be2b3e6774983bf111d2c8daa25ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:25 UTC) from {username='slave'} at 127.0.0.1:57132
I20260924 04:03:25.015185 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7be2b3e6774983bf111d2c8daa25ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:25 UTC) from {username='slave'} at 127.0.0.1:40442
I20260924 04:03:25.015627 5522 tablet_replica.cc:330] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672: stopping tablet replica
I20260924 04:03:25.015821 5285 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7be2b3e6774983bf111d2c8daa25ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:25 UTC) from {username='slave'} at 127.0.0.1:35694
I20260924 04:03:25.016149 5522 raft_consensus.cc:2229] T 1f7be2b3e6774983bf111d2c8daa25ed P 55a4d588d16f40f5b04a53e9c9262672 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:25.016227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5092
I20260924 04:03:25.016952 5524 tablet_replica.cc:330] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: stopping tablet replica
I20260924 04:03:25.017263 5524 raft_consensus.cc:2229] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:25.017592 5524 raft_consensus.cc:2258] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:25.018958 5524 ts_tablet_manager.cc:1905] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:25.021210 5524 ts_tablet_manager.cc:1918] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:25.021297 5524 log.cc:1196] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/1f7be2b3e6774983bf111d2c8daa25ed
I20260924 04:03:25.021525 5523 tablet_replica.cc:330] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: stopping tablet replica
I20260924 04:03:25.021561 5524 ts_tablet_manager.cc:1939] T 1f7be2b3e6774983bf111d2c8daa25ed P 7752547d502849f5895a5bb506bb45e1: Deleting consensus metadata
I20260924 04:03:25.021878 5523 raft_consensus.cc:2229] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:25.022219 5523 raft_consensus.cc:2258] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:03:25.022482 5017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42897 (error 108)
I20260924 04:03:25.022742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5223
I20260924 04:03:25.023593 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:25.023337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42897 (local address 127.0.0.1:57138)
0924 04:03:25.023454 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:25.023521 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 04:03:25.023736 5523 ts_tablet_manager.cc:1905] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:03:25.023804 5017 catalog_manager.cc:4584] TS 55a4d588d16f40f5b04a53e9c9262672 (127.0.98.1:42897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f7be2b3e6774983bf111d2c8daa25ed: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42897: connect: Connection refused (error 111)
I20260924 04:03:25.026288 5523 ts_tablet_manager.cc:1918] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:25.026387 5523 log.cc:1196] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/1f7be2b3e6774983bf111d2c8daa25ed
I20260924 04:03:25.026674 5523 ts_tablet_manager.cc:1939] T 1f7be2b3e6774983bf111d2c8daa25ed P 55ff9bd5b640445abfa03e7cff2aa356: Deleting consensus metadata
I20260924 04:03:25.027966 5016 catalog_manager.cc:4841] TS 55ff9bd5b640445abfa03e7cff2aa356 (127.0.98.3:40379): tablet 1f7be2b3e6774983bf111d2c8daa25ed (table test-table [id=458fccfef45a4203b77fff4f3405f31e]) successfully deleted
I20260924 04:03:25.029990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5354
I20260924 04:03:25.030781 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:25.030100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35467 (local address 127.0.0.1:35700)
0924 04:03:25.030658 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:25.030706 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488}
I20260924 04:03:25.034763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5001
2026-09-24T04:03:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1082 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 04:03:25.054994 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:03:25.056622 392 webserver.cc:460] Webserver started at https://127.0.0.1:35117/ using document root <none> and password file <none>
I20260924 04:03:25.056766 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:03:25.057004 392 webserver.cc:460] Webserver started at http://127.0.0.1:34893/ using document root <none> and password file <none>
2026-09-24T04:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:25Z Disabled control of system clock
I20260924 04:03:25.065675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44789
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44789
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35117/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/testchainca.cert with env {}
W20260924 04:03:25.175599 5531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:25.175890 5531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:25.175972 5531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:25.180047 5531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:25.180115 5531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:25.180130 5531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:25.180145 5531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:25.180158 5531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:03:25.180189 5531 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:03:25.180202 5531 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35117/jwks.json
I20260924 04:03:25.185647 5531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35117/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:25.186726 5531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:25.187707 5531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:25.196223 5531 server_base.cc:1034] running on GCE node
W20260924 04:03:25.196234 5539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:25.196184 5536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:25.196184 5537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:25.197039 5531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:25.198153 5531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:25.199342 5531 hybrid_clock.cc:648] HybridClock initialized: now 1790222605199325 us; error 28 us; skew 500 ppm
I20260924 04:03:25.201548 5531 webserver.cc:460] Webserver started at http://127.0.98.62:35375/ using document root <none> and password file <none>
I20260924 04:03:25.202160 5531 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:25.202227 5531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:25.202428 5531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:25.204625 5531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "e4f19e1b8e7e46599c4220c41f2401d7"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "7defe4a59aa5d15b06d35d7acd43eb86"
server_key_iv: "c1bdb89fd81a230a016d3c7adcbfa3df"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.205180 5531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4f19e1b8e7e46599c4220c41f2401d7"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "7defe4a59aa5d15b06d35d7acd43eb86"
server_key_iv: "c1bdb89fd81a230a016d3c7adcbfa3df"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.208920 5531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:03:25.211398 5545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:25.212843 5531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:25.212978 5531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "e4f19e1b8e7e46599c4220c41f2401d7"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "7defe4a59aa5d15b06d35d7acd43eb86"
server_key_iv: "c1bdb89fd81a230a016d3c7adcbfa3df"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.213091 5531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:25.236581 5531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:25.237303 5531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:25.237484 5531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:25.244258 5531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44789
I20260924 04:03:25.244277 5597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44789 every 8 connection(s)
I20260924 04:03:25.245258 5531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:25.248543 5598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:25.251377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5531
I20260924 04:03:25.251513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:25.251809 392 external_mini_cluster.cc:1428] Setting key 57c5ce8fb08ffb712cf97750e769c1ac
I20260924 04:03:25.254866 5598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7: Bootstrap starting.
I20260924 04:03:25.257184 5598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:25.257925 5598 log.cc:824] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:25.260056 5598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7: No bootstrap required, opened a new log
I20260924 04:03:25.261237 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:25.253237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.0.98.62:44789)
0924 04:03:25.253961 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:25.253976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:25.254015 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:03:25.254657 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:25.254675 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:25.255127 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:25.255452 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:25.255946 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.256976 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.257878 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.258449 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.258629 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.259365 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:25.259370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:25.259384 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:25.259824 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:25.259883 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:25.259937 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:25.260025 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:25.260198 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 04:03:25.260486 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":124,"threads_started":1}
I20260924 04:03:25.263653 5598 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f19e1b8e7e46599c4220c41f2401d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44789 } }
I20260924 04:03:25.263885 5598 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:25.263953 5598 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f19e1b8e7e46599c4220c41f2401d7, State: Initialized, Role: FOLLOWER
I20260924 04:03:25.264294 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:03:25.264402 5598 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f19e1b8e7e46599c4220c41f2401d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44789 } }
I20260924 04:03:25.264549 5598 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:25.264613 5598 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:25.264703 5598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:25.265794 5598 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f19e1b8e7e46599c4220c41f2401d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44789 } }
I20260924 04:03:25.266126 5598 leader_election.cc:302] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4f19e1b8e7e46599c4220c41f2401d7; no voters:
I20260924 04:03:25.266420 5598 leader_election.cc:288] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:25.266570 5603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:25.266796 5603 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [term 1 LEADER]: Becoming Leader. State: Replica: e4f19e1b8e7e46599c4220c41f2401d7, State: Running, Role: LEADER
I20260924 04:03:25.267129 5603 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f19e1b8e7e46599c4220c41f2401d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44789 } }
I20260924 04:03:25.267614 5598 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:25.268730 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4f19e1b8e7e46599c4220c41f2401d7. Latest consensus state: current_term: 1 leader_uuid: "e4f19e1b8e7e46599c4220c41f2401d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f19e1b8e7e46599c4220c41f2401d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44789 } } }
I20260924 04:03:25.268857 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:25.269158 5604 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4f19e1b8e7e46599c4220c41f2401d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f19e1b8e7e46599c4220c41f2401d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44789 } } }
I20260924 04:03:25.269270 5604 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:25.269237 5612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:25.273161 5612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:25.279500 5612 catalog_manager.cc:1347] Generated new cluster ID: 68cb55bd65b54466b81cdbb793c6bd24
I20260924 04:03:25.279656 5612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:25.291108 5612 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:25.292194 5612 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:25.299360 5612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4f19e1b8e7e46599c4220c41f2401d7: Generated new TSK 0
I20260924 04:03:25.300132 5612 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:03:25.325526 5628 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 04:03:25.325664 5628 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:03:25.326167 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:25.308196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.0.98.62:44789)
0924 04:03:25.308325 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:25.308330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:25.308557 (+ 227us) server_negotiation.cc:407] Connection header received
0924 04:03:25.308744 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:25.308746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:25.308793 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:25.308878 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:03:25.309161 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.309712 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.310919 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.311132 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.311198 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.311535 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:03:25.325786 (+ 14251us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:03:25.326051 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:03:25.326246 5601 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 04:03:25.327651 5627 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:25.308112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44789 (local address 127.0.0.1:47586)
0924 04:03:25.308448 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:25.308463 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:25.308599 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:25.308912 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:25.308915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:25.308924 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:03:25.309063 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:25.309070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.309813 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.309816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:25.310817 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:25.310827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.311210 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.311230 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:25.311320 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.311375 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:03:25.326047 (+ 14672us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:03:25.326329 (+ 282us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44789: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44789: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 04:03:25.328703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5531
2026-09-24T04:03:25Z chronyd exiting
I20260924 04:03:25.364879 392 test_util.cc:175] -----------------------------------------------
I20260924 04:03:25.364965 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222576786845-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 04:03:25.365909 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:03:25.366714 392 webserver.cc:460] Webserver started at https://127.0.0.1:42243/ using document root <none> and password file <none>
I20260924 04:03:25.366794 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:03:25.366995 392 webserver.cc:460] Webserver started at http://127.0.0.1:46493/ using document root <none> and password file <none>
2026-09-24T04:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:25Z Disabled control of system clock
I20260924 04:03:25.375011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43941
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42243/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/testchainca.cert with env {}
W20260924 04:03:25.484048 5634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:25.484320 5634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:25.484359 5634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:25.488369 5634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:25.488438 5634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:25.488453 5634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:25.488468 5634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:25.488482 5634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:03:25.488497 5634 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:03:25.488515 5634 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42243/jwks.json
I20260924 04:03:25.494000 5634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42243/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:25.495034 5634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:25.496047 5634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:25.504195 5640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:25.504272 5639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:25.504261 5634 server_base.cc:1034] running on GCE node
W20260924 04:03:25.504505 5642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:25.505033 5634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:25.506134 5634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:25.507308 5634 hybrid_clock.cc:648] HybridClock initialized: now 1790222605507291 us; error 31 us; skew 500 ppm
I20260924 04:03:25.509178 5634 webserver.cc:460] Webserver started at http://127.0.98.62:46521/ using document root <none> and password file <none>
I20260924 04:03:25.509719 5634 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:25.509785 5634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:25.509984 5634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:25.512032 5634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "629b5e577d63437183f1c35262c4ce14"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "1614f02b90a868cdffad0c4c1b57738f"
server_key_iv: "5ba9133ac3a230e868b0d89fc19e0925"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.512571 5634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "629b5e577d63437183f1c35262c4ce14"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "1614f02b90a868cdffad0c4c1b57738f"
server_key_iv: "5ba9133ac3a230e868b0d89fc19e0925"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.516095 5634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:25.518393 5648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:25.519665 5634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:25.519778 5634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "629b5e577d63437183f1c35262c4ce14"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "1614f02b90a868cdffad0c4c1b57738f"
server_key_iv: "5ba9133ac3a230e868b0d89fc19e0925"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.519876 5634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:25.535574 5634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:25.536315 5634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:25.536506 5634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:25.543273 5634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43941
I20260924 04:03:25.543334 5700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43941 every 8 connection(s)
I20260924 04:03:25.544265 5634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:25.547432 5701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:25.550822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5634
I20260924 04:03:25.550940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:25.551194 392 external_mini_cluster.cc:1428] Setting key 3c3eda01ba8242e7d5872666317d59a5
I20260924 04:03:25.553434 5701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14: Bootstrap starting.
I20260924 04:03:25.555619 5701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:25.556382 5701 log.cc:824] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:25.558406 5701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14: No bootstrap required, opened a new log
I20260924 04:03:25.560441 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:25.552575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.0.98.62:43941)
0924 04:03:25.553238 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:25.553253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:25.553301 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:03:25.553974 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:25.553994 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:25.554402 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:25.554752 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:25.555183 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.556227 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.557266 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.557732 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.557873 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.558725 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:25.558729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:25.558742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:25.559132 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:25.559186 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:25.559205 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:25.559288 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:25.559417 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 04:03:25.559671 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260924 04:03:25.562222 5701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629b5e577d63437183f1c35262c4ce14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } }
I20260924 04:03:25.562490 5701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:25.562554 5701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629b5e577d63437183f1c35262c4ce14, State: Initialized, Role: FOLLOWER
I20260924 04:03:25.563014 5701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629b5e577d63437183f1c35262c4ce14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } }
I20260924 04:03:25.563179 5701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:25.563242 5701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:25.563344 5701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:25.563500 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:03:25.564536 5701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629b5e577d63437183f1c35262c4ce14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } }
I20260924 04:03:25.564900 5701 leader_election.cc:302] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 629b5e577d63437183f1c35262c4ce14; no voters:
I20260924 04:03:25.565253 5701 leader_election.cc:288] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:25.565361 5706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:25.565608 5706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [term 1 LEADER]: Becoming Leader. State: Replica: 629b5e577d63437183f1c35262c4ce14, State: Running, Role: LEADER
I20260924 04:03:25.565943 5706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629b5e577d63437183f1c35262c4ce14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } }
I20260924 04:03:25.566462 5701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:25.567440 5708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 629b5e577d63437183f1c35262c4ce14. Latest consensus state: current_term: 1 leader_uuid: "629b5e577d63437183f1c35262c4ce14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629b5e577d63437183f1c35262c4ce14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } } }
I20260924 04:03:25.567574 5708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:25.567840 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "629b5e577d63437183f1c35262c4ce14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629b5e577d63437183f1c35262c4ce14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } } }
I20260924 04:03:25.567945 5715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:25.567968 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:25.571323 5715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:25.577067 5715 catalog_manager.cc:1347] Generated new cluster ID: 2f522f7179364b8388b8f6144295c917
I20260924 04:03:25.577159 5715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:25.586699 5715 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:25.587954 5715 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:25.593326 5715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 629b5e577d63437183f1c35262c4ce14: Generated new TSK 0
I20260924 04:03:25.594031 5715 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:03:25.620004 5731 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 04:03:25.620100 5731 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:03:25.620786 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:25.604881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.0.98.62:43941)
0924 04:03:25.605020 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:25.605024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:25.605181 (+ 157us) server_negotiation.cc:407] Connection header received
0924 04:03:25.605331 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:25.605333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:25.605387 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:25.605489 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:03:25.605742 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.606313 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.607582 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.607809 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.607892 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.608273 (+ 381us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:03:25.620408 (+ 12135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:03:25.620668 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:03:25.620850 5704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 04:03:25.621063 5730 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:25.604735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43941 (local address 127.0.0.1:54858)
0924 04:03:25.605053 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:25.605070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:25.605202 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:25.605505 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:25.605509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:25.605517 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:03:25.605644 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:25.605652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.606423 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.606427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:25.607440 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:25.607453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.607933 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.607955 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:25.608058 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.608091 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:03:25.620645 (+ 12554us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:03:25.620861 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42243/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":112,"threads_started":1}
I20260924 04:03:25.621644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5634
2026-09-24T04:03:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (294 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 04:03:25.660384 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:03:25.661279 392 webserver.cc:460] Webserver started at https://127.0.0.1:37363/ using document root <none> and password file <none>
I20260924 04:03:25.661378 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:03:25.661645 392 webserver.cc:460] Webserver started at http://127.0.0.1:36097/ using document root <none> and password file <none>
2026-09-24T04:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:25Z Disabled control of system clock
I20260924 04:03:25.670251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39583
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39583
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37363/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:03:25.780855 5737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:25.781165 5737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:25.781206 5737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:25.785318 5737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:25.785388 5737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:25.785422 5737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:25.785439 5737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:25.785452 5737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:03:25.785467 5737 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:03:25.785486 5737 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37363/jwks.json
I20260924 04:03:25.790789 5737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37363/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:25.791882 5737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:25.793280 5737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:25.801509 5742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:25.801607 5737 server_base.cc:1034] running on GCE node
W20260924 04:03:25.801509 5745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:25.801537 5743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:25.802309 5737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:25.803401 5737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:25.804581 5737 hybrid_clock.cc:648] HybridClock initialized: now 1790222605804554 us; error 38 us; skew 500 ppm
I20260924 04:03:25.806490 5737 webserver.cc:460] Webserver started at http://127.0.98.62:46651/ using document root <none> and password file <none>
I20260924 04:03:25.807055 5737 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:25.807120 5737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:25.807307 5737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:25.809362 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "951fecccf55e4aa5a83db640aa2c8bc0"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "4bb4a63a8f9d06c29e17c0e67e25d180"
server_key_iv: "46a216df4ddad4e5c4b0f9adf4f3e035"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.809906 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "951fecccf55e4aa5a83db640aa2c8bc0"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "4bb4a63a8f9d06c29e17c0e67e25d180"
server_key_iv: "46a216df4ddad4e5c4b0f9adf4f3e035"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.813544 5737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:03:25.816219 5751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:25.817523 5737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:03:25.817647 5737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "951fecccf55e4aa5a83db640aa2c8bc0"
format_stamp: "Formatted at 2026-09-24 04:03:25 on dist-test-slave-zcsk"
server_key: "4bb4a63a8f9d06c29e17c0e67e25d180"
server_key_iv: "46a216df4ddad4e5c4b0f9adf4f3e035"
server_key_version: "encryptionkey@0"
I20260924 04:03:25.817750 5737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:25.838042 5737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:25.838809 5737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:25.838995 5737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:25.845849 5737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39583
I20260924 04:03:25.845891 5803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39583 every 8 connection(s)
I20260924 04:03:25.846827 5737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:25.849970 5804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:25.855439 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0: Bootstrap starting.
I20260924 04:03:25.856078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5737
I20260924 04:03:25.856201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:25.856463 392 external_mini_cluster.cc:1428] Setting key 619e8c10a5b72ce8b43deacc540ffbaa
I20260924 04:03:25.858322 5804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:25.859071 5804 log.cc:824] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:25.861145 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0: No bootstrap required, opened a new log
I20260924 04:03:25.864723 5804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951fecccf55e4aa5a83db640aa2c8bc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39583 } }
I20260924 04:03:25.864995 5804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:25.865051 5804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951fecccf55e4aa5a83db640aa2c8bc0, State: Initialized, Role: FOLLOWER
I20260924 04:03:25.865494 5804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951fecccf55e4aa5a83db640aa2c8bc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39583 } }
I20260924 04:03:25.865615 5804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:25.865672 5804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:25.865737 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:25.857769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.0.98.62:39583)
0924 04:03:25.858506 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:25.858523 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:25.858561 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:03:25.859259 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:25.859282 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:25.859792 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:25.860203 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:25.860633 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.861645 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.862537 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.863005 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.863146 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.863898 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:25.863930 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:25.863949 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:25.864350 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:25.864405 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:25.864424 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:25.864516 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:25.864652 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 04:03:25.864905 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":136,"threads_started":1}
I20260924 04:03:25.865758 5804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:25.866781 5804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951fecccf55e4aa5a83db640aa2c8bc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39583 } }
I20260924 04:03:25.867132 5804 leader_election.cc:302] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 951fecccf55e4aa5a83db640aa2c8bc0; no voters:
I20260924 04:03:25.867457 5804 leader_election.cc:288] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:25.867601 5809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:25.867815 5809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 951fecccf55e4aa5a83db640aa2c8bc0, State: Running, Role: LEADER
I20260924 04:03:25.868151 5809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951fecccf55e4aa5a83db640aa2c8bc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39583 } }
I20260924 04:03:25.868664 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:03:25.868741 5804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:25.870000 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 951fecccf55e4aa5a83db640aa2c8bc0. Latest consensus state: current_term: 1 leader_uuid: "951fecccf55e4aa5a83db640aa2c8bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951fecccf55e4aa5a83db640aa2c8bc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39583 } } }
I20260924 04:03:25.870113 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:25.870433 5811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "951fecccf55e4aa5a83db640aa2c8bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951fecccf55e4aa5a83db640aa2c8bc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39583 } } }
I20260924 04:03:25.870532 5811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0 [sys.catalog]: This master's current role is: LEADER
W20260924 04:03:25.873142 5824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:03:25.873204 5824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:03:25.873323 5820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:25.874172 5820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:25.879439 5820 catalog_manager.cc:1347] Generated new cluster ID: 48bbebfd1e044fdd8d6cb8c37d99fbf6
I20260924 04:03:25.879505 5820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:25.906204 5820 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:25.907157 5820 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:25.913304 5820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 951fecccf55e4aa5a83db640aa2c8bc0: Generated new TSK 0
I20260924 04:03:25.914004 5820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:03:25.934232 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:25.929078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.0.98.62:39583)
0924 04:03:25.929213 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:25.929218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:25.929380 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:03:25.929550 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:25.929552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:25.929600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:25.929678 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:03:25.929892 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.930419 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.931612 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.931824 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.931895 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.932217 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:03:25.933872 (+ 1655us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:03:25.934096 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:03:25.934314 5833 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:25.928982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39583 (local address 127.0.0.1:33038)
0924 04:03:25.929300 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:25.929315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:25.929429 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:25.929694 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:25.929697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:25.929704 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:03:25.929808 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:25.929815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.930511 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.930514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:25.931507 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:25.931517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:25.931940 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:25.931959 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:25.932047 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:25.932074 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:03:25.934026 (+ 1952us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:03:25.934161 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260924 04:03:25.934643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5737
W20260924 04:03:25.934643 5807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37363/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
E20260924 04:03:25.945012 5834 webserver.cc:533] Webserver: sslize failed:
E20260924 04:03:25.945091 5834 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T04:03:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T04:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:25Z Disabled control of system clock
I20260924 04:03:25.980247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45649
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45649
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/testchainca.cert with env {}
W20260924 04:03:26.089382 5838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:26.089664 5838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:26.089704 5838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:26.093758 5838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:26.093828 5838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:26.093842 5838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:26.093858 5838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:26.093873 5838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:03:26.093887 5838 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:03:26.093904 5838 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 04:03:26.099380 5838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:26.100534 5838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:26.101547 5838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:26.109748 5846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:26.109802 5843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:26.109776 5844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:26.110014 5838 server_base.cc:1034] running on GCE node
I20260924 04:03:26.110636 5838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:26.111757 5838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:26.112970 5838 hybrid_clock.cc:648] HybridClock initialized: now 1790222606112941 us; error 46 us; skew 500 ppm
I20260924 04:03:26.115051 5838 webserver.cc:460] Webserver started at http://127.0.98.62:38245/ using document root <none> and password file <none>
I20260924 04:03:26.115623 5838 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:26.115696 5838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:26.115939 5838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:26.118006 5838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4"
format_stamp: "Formatted at 2026-09-24 04:03:26 on dist-test-slave-zcsk"
server_key: "4bc1ca7f04857b51cb1b0c9708b63ae9"
server_key_iv: "ab047a8ce22b676d984bd0621e414ad3"
server_key_version: "encryptionkey@0"
I20260924 04:03:26.118589 5838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4"
format_stamp: "Formatted at 2026-09-24 04:03:26 on dist-test-slave-zcsk"
server_key: "4bc1ca7f04857b51cb1b0c9708b63ae9"
server_key_iv: "ab047a8ce22b676d984bd0621e414ad3"
server_key_version: "encryptionkey@0"
I20260924 04:03:26.122645 5838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:03:26.125077 5852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:26.126389 5838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:26.126516 5838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4"
format_stamp: "Formatted at 2026-09-24 04:03:26 on dist-test-slave-zcsk"
server_key: "4bc1ca7f04857b51cb1b0c9708b63ae9"
server_key_iv: "ab047a8ce22b676d984bd0621e414ad3"
server_key_version: "encryptionkey@0"
I20260924 04:03:26.126619 5838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:26.138496 5838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:26.139201 5838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:26.139387 5838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:26.145979 5838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45649
I20260924 04:03:26.145993 5904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45649 every 8 connection(s)
I20260924 04:03:26.146911 5838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:26.149773 5905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:26.155117 5905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4: Bootstrap starting.
I20260924 04:03:26.155469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5838
I20260924 04:03:26.155570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:26.155817 392 external_mini_cluster.cc:1428] Setting key 61ebe0552eaf517be13126bd229c10c3
I20260924 04:03:26.157675 5905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:26.158380 5905 log.cc:824] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:26.160295 5905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4: No bootstrap required, opened a new log
I20260924 04:03:26.163620 5905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } }
I20260924 04:03:26.163846 5905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:26.163893 5905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cb95486a9f4fcaa9b6ab6a655eb2d4, State: Initialized, Role: FOLLOWER
I20260924 04:03:26.164356 5905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } }
I20260924 04:03:26.164477 5905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:26.164535 5905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:26.164623 5905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:26.164852 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:26.157073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.0.98.62:45649)
0924 04:03:26.157729 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:26.157746 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:26.157787 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:03:26.158448 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:26.158464 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:26.158969 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:26.159331 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:26.159759 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:26.160797 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:26.161673 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:26.162147 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:26.162293 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:26.163032 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:26.163037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:26.163049 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:26.163446 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:26.163501 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:26.163521 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:26.163600 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:26.163736 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 04:03:26.164059 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":129,"threads_started":1}
I20260924 04:03:26.165508 5905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } }
I20260924 04:03:26.165813 5905 leader_election.cc:302] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45cb95486a9f4fcaa9b6ab6a655eb2d4; no voters:
I20260924 04:03:26.166107 5905 leader_election.cc:288] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:26.166191 5910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:26.166415 5910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [term 1 LEADER]: Becoming Leader. State: Replica: 45cb95486a9f4fcaa9b6ab6a655eb2d4, State: Running, Role: LEADER
I20260924 04:03:26.166998 5910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } }
I20260924 04:03:26.167325 5905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:26.167613 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:03:26.168713 5911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } } }
I20260924 04:03:26.168846 5911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:26.169246 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45cb95486a9f4fcaa9b6ab6a655eb2d4. Latest consensus state: current_term: 1 leader_uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cb95486a9f4fcaa9b6ab6a655eb2d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45649 } } }
I20260924 04:03:26.169413 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4 [sys.catalog]: This master's current role is: LEADER
W20260924 04:03:26.171983 5925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:03:26.172044 5925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:03:26.172112 5923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:26.172863 5923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:26.178063 5923 catalog_manager.cc:1347] Generated new cluster ID: d478d63c2fc146b5b93b7fbde6e2240f
I20260924 04:03:26.178133 5923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:26.188230 5923 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:26.189495 5923 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:26.203719 5923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45cb95486a9f4fcaa9b6ab6a655eb2d4: Generated new TSK 0
I20260924 04:03:26.204591 5923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:03:26.271615 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:26.223672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.0.98.62:45649)
0924 04:03:26.223807 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:26.223811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:26.223986 (+ 175us) server_negotiation.cc:407] Connection header received
0924 04:03:26.224136 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:26.224139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:26.224195 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:26.224322 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:03:26.224556 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:26.225105 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:26.226351 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:26.226584 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:26.226677 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:26.227036 (+ 359us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:03:26.271217 (+ 44181us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:03:26.271494 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59992: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 04:03:26.271695 5908 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59992: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 04:03:26.271934 5934 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:26.223525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45649 (local address 127.0.0.1:59992)
0924 04:03:26.223852 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:26.223868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:26.224018 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:26.224317 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:26.224321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:26.224330 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:03:26.224452 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:26.224459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:26.225207 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:26.225210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:26.226209 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:26.226220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:26.226678 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:26.226703 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:26.226792 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:26.226822 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:03:26.271464 (+ 44642us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:03:26.271702 (+ 238us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":129,"threads_started":1}
I20260924 04:03:26.272562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5838
2026-09-24T04:03:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (317 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:26 dist-test-slave-zcsk krb5kdc[5946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:26 dist-test-slave-zcsk krb5kdc[5946](info): set up 2 sockets
Sep 24 04:03:26 dist-test-slave-zcsk krb5kdc[5946](info): commencing operation
krb5kdc: starting...
W20260924 04:03:27.464025 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:27 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:27.578199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34139
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:27.686221 5959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:27.686511 5959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:27.686551 5959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:27.690704 5959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:27.690788 5959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:27.690806 5959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:27.690821 5959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:27.690836 5959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:27.696342 5959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:27.697444 5959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:27.698477 5959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:27.707065 5964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:27.707065 5967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:27.707059 5965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:27.707392 5959 server_base.cc:1034] running on GCE node
I20260924 04:03:27.707939 5959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:27.709096 5959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:27.710304 5959 hybrid_clock.cc:648] HybridClock initialized: now 1790222607710271 us; error 50 us; skew 500 ppm
Sep 24 04:03:27 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:27.713038 5959 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:27.714033 5959 webserver.cc:460] Webserver started at http://127.0.98.62:37391/ using document root <none> and password file <none>
I20260924 04:03:27.714593 5959 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:27.714659 5959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:27.714855 5959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:27.717036 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "87b9b80a93c346be81f6b2413376d23c"
format_stamp: "Formatted at 2026-09-24 04:03:27 on dist-test-slave-zcsk"
server_key: "93cdb0d9542ea4a7e73a9695367e2fda"
server_key_iv: "674f1319db144fb618901be66943001b"
server_key_version: "encryptionkey@0"
I20260924 04:03:27.717612 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "87b9b80a93c346be81f6b2413376d23c"
format_stamp: "Formatted at 2026-09-24 04:03:27 on dist-test-slave-zcsk"
server_key: "93cdb0d9542ea4a7e73a9695367e2fda"
server_key_iv: "674f1319db144fb618901be66943001b"
server_key_version: "encryptionkey@0"
I20260924 04:03:27.721549 5959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:03:27.724131 5974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:27.725405 5959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:27.725528 5959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "87b9b80a93c346be81f6b2413376d23c"
format_stamp: "Formatted at 2026-09-24 04:03:27 on dist-test-slave-zcsk"
server_key: "93cdb0d9542ea4a7e73a9695367e2fda"
server_key_iv: "674f1319db144fb618901be66943001b"
server_key_version: "encryptionkey@0"
I20260924 04:03:27.725637 5959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:27.737432 5959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:27.740547 5959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:27.740727 5959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:27.747601 5959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34139
I20260924 04:03:27.747845 6026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34139 every 8 connection(s)
I20260924 04:03:27.748631 5959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:27.751433 6027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:27.754225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 5959
I20260924 04:03:27.754343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:27.754576 392 external_mini_cluster.cc:1428] Setting key b9e79af37e048e8dcd10bcbf1c5405f0
I20260924 04:03:27.757254 6027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c: Bootstrap starting.
I20260924 04:03:27.759714 6027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:27.760637 6027 log.cc:824] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:27.762838 6027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c: No bootstrap required, opened a new log
Sep 24 04:03:27 dist-test-slave-zcsk krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:27.765980 6027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9b80a93c346be81f6b2413376d23c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34139 } }
I20260924 04:03:27.766215 6027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:27.766248 6027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b9b80a93c346be81f6b2413376d23c, State: Initialized, Role: FOLLOWER
I20260924 04:03:27.766755 6027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9b80a93c346be81f6b2413376d23c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34139 } }
I20260924 04:03:27.766912 6027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:27.766979 6027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:27.767072 6027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:27.768280 6027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9b80a93c346be81f6b2413376d23c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34139 } }
I20260924 04:03:27.768673 6027 leader_election.cc:302] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b9b80a93c346be81f6b2413376d23c; no voters:
I20260924 04:03:27.769016 6027 leader_election.cc:288] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:27.769140 6032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:27.769368 6032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [term 1 LEADER]: Becoming Leader. State: Replica: 87b9b80a93c346be81f6b2413376d23c, State: Running, Role: LEADER
I20260924 04:03:27.769682 6032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9b80a93c346be81f6b2413376d23c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34139 } }
I20260924 04:03:27.769865 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:27.755892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.98.62:34139)
0924 04:03:27.756324 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:27.756341 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:27.756387 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:03:27.757009 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:27.757026 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:27.757375 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:27.757679 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:27.758537 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:27.759507 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:27.760656 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:27.761260 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:27.761412 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:27.764121 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:27.764126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:27.764140 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:27.764170 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:27.766349 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:27.766846 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:27.766851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:27.766855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:27.767001 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:27.767308 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:27.767311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:27.767313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:27.767847 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:27.768077 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:27.768392 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 04:03:27.768712 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260924 04:03:27.770358 6027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:27.770956 6033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b9b80a93c346be81f6b2413376d23c. Latest consensus state: current_term: 1 leader_uuid: "87b9b80a93c346be81f6b2413376d23c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9b80a93c346be81f6b2413376d23c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34139 } } }
I20260924 04:03:27.771080 6033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:27.771724 6034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b9b80a93c346be81f6b2413376d23c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9b80a93c346be81f6b2413376d23c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34139 } } }
I20260924 04:03:27.771821 6034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:27.772377 6039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:27.775532 6039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:27.781054 6039 catalog_manager.cc:1347] Generated new cluster ID: 8adbe9f835a24efb92af36158d274ac8
I20260924 04:03:27.781140 6039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:27.798834 6039 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:27.800215 6039 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:27.812815 6039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b9b80a93c346be81f6b2413376d23c: Generated new TSK 0
I20260924 04:03:27.813589 6039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:27.847312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34139
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:27.958606 6053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:27.958891 6053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:27.958935 6053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:27.963100 6053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:27.963171 6053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:27.963254 6053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:27.968717 6053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:27.969817 6053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:27.970920 6053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:27.980952 6061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:27.980952 6058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:27.981098 6059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:27.981231 6053 server_base.cc:1034] running on GCE node
I20260924 04:03:27.981771 6053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:27.982517 6053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:27.983691 6053 hybrid_clock.cc:648] HybridClock initialized: now 1790222607983663 us; error 43 us; skew 500 ppm
Sep 24 04:03:27 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:27.986487 6053 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:27.987403 6053 webserver.cc:460] Webserver started at http://127.0.98.1:46459/ using document root <none> and password file <none>
I20260924 04:03:27.987982 6053 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:27.988050 6053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:27.988224 6053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:27.990280 6053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "0cd67e6f21584aa69391b043d574fd48"
format_stamp: "Formatted at 2026-09-24 04:03:27 on dist-test-slave-zcsk"
server_key: "12a0c75a1396245c6decef85be7ad6a6"
server_key_iv: "7ba487bc9624006a5e1e6dbd5f92177b"
server_key_version: "encryptionkey@0"
I20260924 04:03:27.990793 6053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0cd67e6f21584aa69391b043d574fd48"
format_stamp: "Formatted at 2026-09-24 04:03:27 on dist-test-slave-zcsk"
server_key: "12a0c75a1396245c6decef85be7ad6a6"
server_key_iv: "7ba487bc9624006a5e1e6dbd5f92177b"
server_key_version: "encryptionkey@0"
I20260924 04:03:27.994617 6053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:03:27.997387 6068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:27.998677 6053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:27.998796 6053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "0cd67e6f21584aa69391b043d574fd48"
format_stamp: "Formatted at 2026-09-24 04:03:27 on dist-test-slave-zcsk"
server_key: "12a0c75a1396245c6decef85be7ad6a6"
server_key_iv: "7ba487bc9624006a5e1e6dbd5f92177b"
server_key_version: "encryptionkey@0"
I20260924 04:03:27.998906 6053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:28.013103 6053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:28.017946 6053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:28.018198 6053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:28.018976 6053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:28.020104 6053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:28.020184 6053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:28.020262 6053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:28.020300 6053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:28.030043 6053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42717
I20260924 04:03:28.030083 6181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42717 every 8 connection(s)
I20260924 04:03:28.031002 6053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:28.033418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 6053
I20260924 04:03:28.033525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:28.033806 392 external_mini_cluster.cc:1428] Setting key 388aed7039bc0e7647c6c5af9450fc8c
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:03:28.045257 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.032799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56287 (local address 127.0.98.62:34139)
0924 04:03:28.032946 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.032950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.033827 (+ 877us) server_negotiation.cc:407] Connection header received
0924 04:03:28.035439 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.035443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.035491 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.035577 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.036844 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.037617 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.038731 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.038959 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.039045 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.041687 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:28.041690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:28.041692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:28.041716 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:28.043126 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.043605 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.043608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.043609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.043682 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.043970 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.043972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.043973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.044157 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:28.044236 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:28.044951 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 04:03:28.045085 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:03:28.046044 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.033203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34139 (local address 127.0.98.1:56287)
0924 04:03:28.033658 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:28.033711 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:28.035214 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:28.035719 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:28.035725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:28.036181 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:28.036704 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:28.036717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.037742 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.037744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:28.038614 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:28.038622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.039094 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.039098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:28.039255 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.039818 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:28.039834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:28.041517 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:28.043250 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:28.043256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:28.043270 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:28.043526 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:28.043747 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:28.043749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:28.043750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:28.043884 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:28.044285 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:28.044292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:28.044636 (+ 344us) client_negotiation.cc:769] Sending connection context
0924 04:03:28.044841 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 04:03:28.045125 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260924 04:03:28.047287 6182 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34139
I20260924 04:03:28.047518 6182 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:28.048030 6182 heartbeater.cc:511] Master 127.0.98.62:34139 requested a full tablet report, sending...
I20260924 04:03:28.049614 5991 ts_manager.cc:194] Registered new tserver with Master: 0cd67e6f21584aa69391b043d574fd48 (127.0.98.1:42717)
I20260924 04:03:28.051157 5991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:03:28.058732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34139
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:28.168629 6187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:28.168931 6187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:28.168993 6187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:28.173218 6187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:28.173296 6187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:28.173381 6187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:28.178939 6187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:28.180074 6187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:28.181109 6187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:28.190779 6195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:28.190814 6187 server_base.cc:1034] running on GCE node
W20260924 04:03:28.190755 6192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:28.190755 6193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:28.191428 6187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:28.192106 6187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:28.193288 6187 hybrid_clock.cc:648] HybridClock initialized: now 1790222608193265 us; error 32 us; skew 500 ppm
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.196007 6187 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:28.196902 6187 webserver.cc:460] Webserver started at http://127.0.98.2:43947/ using document root <none> and password file <none>
I20260924 04:03:28.197438 6187 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:28.197502 6187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:28.197685 6187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:28.199738 6187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "e3a10bd7eacf491b865f03f7640f7b54"
format_stamp: "Formatted at 2026-09-24 04:03:28 on dist-test-slave-zcsk"
server_key: "64bf02909d6c5c611d64bab21fdaa159"
server_key_iv: "c1a85196b990ae29abb956fa02cfb6e0"
server_key_version: "encryptionkey@0"
I20260924 04:03:28.200343 6187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e3a10bd7eacf491b865f03f7640f7b54"
format_stamp: "Formatted at 2026-09-24 04:03:28 on dist-test-slave-zcsk"
server_key: "64bf02909d6c5c611d64bab21fdaa159"
server_key_iv: "c1a85196b990ae29abb956fa02cfb6e0"
server_key_version: "encryptionkey@0"
I20260924 04:03:28.204360 6187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:03:28.206923 6202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:28.208312 6187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:28.208426 6187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "e3a10bd7eacf491b865f03f7640f7b54"
format_stamp: "Formatted at 2026-09-24 04:03:28 on dist-test-slave-zcsk"
server_key: "64bf02909d6c5c611d64bab21fdaa159"
server_key_iv: "c1a85196b990ae29abb956fa02cfb6e0"
server_key_version: "encryptionkey@0"
I20260924 04:03:28.208531 6187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:28.230262 6187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:28.233536 6187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:28.233721 6187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:28.234323 6187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:28.235157 6187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:28.235213 6187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:28.235267 6187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:28.235292 6187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:28.243643 6187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43863
I20260924 04:03:28.243667 6315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43863 every 8 connection(s)
I20260924 04:03:28.244586 6187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:28.244828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 6187
I20260924 04:03:28.244939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:28.245240 392 external_mini_cluster.cc:1428] Setting key 4e9528bab746764b374e909835f08b73
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:28.261931 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.246520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32987 (local address 127.0.98.62:34139)
0924 04:03:28.246642 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.246645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.247636 (+ 991us) server_negotiation.cc:407] Connection header received
0924 04:03:28.249055 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.249058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.249106 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.249186 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.250789 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.251334 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.253114 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.253344 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.253430 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.258323 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:28.258327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:28.258329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:28.258357 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:28.259660 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.260312 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.260315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.260317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.260403 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.260752 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.260755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.260756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.260952 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:28.261037 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:28.261613 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 04:03:28.261755 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:03:28.262712 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.246995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34139 (local address 127.0.98.2:32987)
0924 04:03:28.247463 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:28.247517 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:28.248838 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:28.249408 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:28.249418 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:28.249902 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:28.250623 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:28.250646 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.251479 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.251483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:28.252979 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:28.252990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.253503 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.253507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:28.253713 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.254334 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:28.254355 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:28.258074 (+ 3719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:28.259795 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:28.259801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:28.259816 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:28.260189 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:28.260486 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:28.260490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:28.260492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:28.260649 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:28.261050 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:28.261057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:28.261327 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 04:03:28.261509 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:03:28.261810 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260924 04:03:28.264086 6316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34139
I20260924 04:03:28.264339 6316 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:28.264840 6316 heartbeater.cc:511] Master 127.0.98.62:34139 requested a full tablet report, sending...
I20260924 04:03:28.266026 5991 ts_manager.cc:194] Registered new tserver with Master: e3a10bd7eacf491b865f03f7640f7b54 (127.0.98.2:43863)
I20260924 04:03:28.266625 5991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:03:28.272997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34139
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:28.384622 6321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:28.384895 6321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:28.384943 6321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:28.389195 6321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:28.389258 6321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:28.389329 6321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:28.394711 6321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:28.395833 6321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:28.396885 6321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:28.406241 6329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:28.406239 6326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:28.406242 6327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:28.406323 6321 server_base.cc:1034] running on GCE node
I20260924 04:03:28.406880 6321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:28.407531 6321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:28.408689 6321 hybrid_clock.cc:648] HybridClock initialized: now 1790222608408668 us; error 33 us; skew 500 ppm
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.411335 6321 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:28.412233 6321 webserver.cc:460] Webserver started at http://127.0.98.3:42023/ using document root <none> and password file <none>
I20260924 04:03:28.412817 6321 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:28.412886 6321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:28.413072 6321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:28.415055 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "57bf55298f37475db1b85415e432f808"
format_stamp: "Formatted at 2026-09-24 04:03:28 on dist-test-slave-zcsk"
server_key: "bafe2c50fdfd11dee24c97e8235c4ee1"
server_key_iv: "1629b8fb7f7dd44f2fc02b4ef671ecff"
server_key_version: "encryptionkey@0"
I20260924 04:03:28.415570 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "57bf55298f37475db1b85415e432f808"
format_stamp: "Formatted at 2026-09-24 04:03:28 on dist-test-slave-zcsk"
server_key: "bafe2c50fdfd11dee24c97e8235c4ee1"
server_key_iv: "1629b8fb7f7dd44f2fc02b4ef671ecff"
server_key_version: "encryptionkey@0"
I20260924 04:03:28.419312 6321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:03:28.421754 6336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:28.423053 6321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:28.423211 6321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "57bf55298f37475db1b85415e432f808"
format_stamp: "Formatted at 2026-09-24 04:03:28 on dist-test-slave-zcsk"
server_key: "bafe2c50fdfd11dee24c97e8235c4ee1"
server_key_iv: "1629b8fb7f7dd44f2fc02b4ef671ecff"
server_key_version: "encryptionkey@0"
I20260924 04:03:28.423360 6321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:28.435269 6321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:28.438191 6321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:28.438375 6321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:28.438941 6321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:28.439743 6321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:28.439790 6321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:28.439855 6321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:28.439884 6321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:28.448060 6321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45823
I20260924 04:03:28.448081 6449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45823 every 8 connection(s)
I20260924 04:03:28.448948 6321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:28.459064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 6321
I20260924 04:03:28.459199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:28.459486 392 external_mini_cluster.cc:1428] Setting key 90d4067ad7d73bf4c866bdc2097664cb
I20260924 04:03:28.461639 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.450695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40729 (local address 127.0.98.62:34139)
0924 04:03:28.450858 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.450862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.451657 (+ 795us) server_negotiation.cc:407] Connection header received
0924 04:03:28.453015 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.453018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.453063 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.453124 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.454220 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.454740 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.455769 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.456016 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.456087 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.458334 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:28.458337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:28.458339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:28.458361 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:28.459628 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.460179 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.460183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.460184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.460312 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.460665 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.460668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.460670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.460879 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:28.460966 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:28.461422 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 04:03:28.461530 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:03:28.462445 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.451094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34139 (local address 127.0.98.3:40729)
0924 04:03:28.451512 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:28.451554 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:28.452805 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:28.453300 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:28.453307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:28.453646 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:28.454115 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:28.454128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.454824 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.454826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:28.455685 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:28.455693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.456123 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.456126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:28.456258 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.456817 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:28.456832 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:28.458178 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:28.459723 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:28.459728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:28.459742 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:28.460095 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:28.460402 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:28.460405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:28.460407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:28.460578 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:28.460970 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:28.460976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:28.461196 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 04:03:28.461369 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 04:03:28.461657 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260924 04:03:28.463387 6450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34139
I20260924 04:03:28.463586 6450 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:28.464048 6450 heartbeater.cc:511] Master 127.0.98.62:34139 requested a full tablet report, sending...
I20260924 04:03:28.465018 5991 ts_manager.cc:194] Registered new tserver with Master: 57bf55298f37475db1b85415e432f808 (127.0.98.3:45823)
I20260924 04:03:28.465600 5991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40729
I20260924 04:03:28.473117 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/test-admin.keytab.
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.488013 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 04:03:28.490717 6457 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:28.489343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.98.1:42717)
0924 04:03:28.489681 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.489693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.489721 (+ 28us) server_negotiation.cc:407] Connection header received
0924 04:03:28.489772 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.489778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.489986 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.490115 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.490534 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43650: BlockingRecv error: recv got EOF from 127.0.0.1:43650 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":133,"threads_started":1}
W20260924 04:03:28.490836 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.491842 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.492731 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:03:28.501991 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.493600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.98.1:42717)
0924 04:03:28.493742 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.493746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.493762 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:03:28.493928 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.493931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.493976 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.494056 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.494567 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.495374 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.496246 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.496625 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.496705 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.498845 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:28.498854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:28.498867 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:28.498891 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:28.500399 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.500784 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.500790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.500791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.500863 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.501093 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.501096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.501098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.501347 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:28.501551 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:28.501716 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:03:28.501857 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 04:03:28.505797 6457 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:28.504902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.0.98.1:42717)
0924 04:03:28.505025 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.505029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.505113 (+ 84us) server_negotiation.cc:407] Connection header received
0924 04:03:28.505248 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.505250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.505284 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.505343 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.505696 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43668: BlockingRecv error: recv got EOF from 127.0.0.1:43668 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:03:28.505909 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.506886 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.507678 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:03:28.515509 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.508485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.98.1:42717)
0924 04:03:28.508600 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.508603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.508669 (+ 66us) server_negotiation.cc:407] Connection header received
0924 04:03:28.508782 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.508785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.508819 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.508872 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.509302 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.509806 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.510506 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.510730 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.510799 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.513003 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:28.513006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:28.513008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:28.513030 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:28.514159 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.514506 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.514509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.514510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.514576 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.514827 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.514830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.514831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.515001 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:28.515096 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:28.515265 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:03:28.515356 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:03:28.517984 6457 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:28.517173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.0.98.1:42717)
0924 04:03:28.517280 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.517284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.517315 (+ 31us) server_negotiation.cc:407] Connection header received
0924 04:03:28.517432 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.517435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.517472 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.517533 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.517897 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43682: BlockingRecv error: recv got EOF from 127.0.0.1:43682 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 04:03:28.518100 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222576786845-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.518996 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:28.519795 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:03:28.527854 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:28.520601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.0.98.1:42717)
0924 04:03:28.520715 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:28.520719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:28.520807 (+ 88us) server_negotiation.cc:407] Connection header received
0924 04:03:28.520918 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:28.520920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:28.520962 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:28.521026 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:28.521481 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.521963 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.522710 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:28.522947 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:28.523029 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:28.525113 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:28.525116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:28.525118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:28.525140 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:28.526398 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.526763 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.526767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.526768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.526840 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:28.527066 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:28.527069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:28.527071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:28.527278 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:28.527391 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:28.527604 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 04:03:28.527704 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:03:28.528831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 6053
I20260924 04:03:28.534113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 6187
I20260924 04:03:28.538664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 6321
I20260924 04:03:28.543412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 5959
2026-09-24T04:03:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2033 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[6470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[6470](info): set up 2 sockets
Sep 24 04:03:28 dist-test-slave-zcsk krb5kdc[6470](info): commencing operation
krb5kdc: starting...
W20260924 04:03:29.500098 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:29 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:03:29.622603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46295
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:29.733250 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:29.733557 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:29.733598 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:29.737712 6483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:29.737783 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:29.737802 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:29.737818 6483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:29.737831 6483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:29.743341 6483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:29.744566 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:29.745625 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:29.754331 6491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:29.754311 6488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:29.754311 6489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:29.754388 6483 server_base.cc:1034] running on GCE node
I20260924 04:03:29.755168 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:29.756397 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:29.757570 6483 hybrid_clock.cc:648] HybridClock initialized: now 1790222609757543 us; error 46 us; skew 500 ppm
Sep 24 04:03:29 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:29.765514 6483 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:03:29.766520 6483 webserver.cc:460] Webserver started at http://127.0.98.62:40655/ using document root <none> and password file <none>
I20260924 04:03:29.767154 6483 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:29.767292 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:29.767603 6483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:29.769893 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "4e2e7e0e64f44abf9ea715aa39cbb843"
format_stamp: "Formatted at 2026-09-24 04:03:29 on dist-test-slave-zcsk"
server_key: "6f5f4e95bf889530a5cbc4b862345791"
server_key_iv: "03cacea5f3909a68c5c30bed038dd225"
server_key_version: "encryptionkey@0"
I20260924 04:03:29.770507 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e2e7e0e64f44abf9ea715aa39cbb843"
format_stamp: "Formatted at 2026-09-24 04:03:29 on dist-test-slave-zcsk"
server_key: "6f5f4e95bf889530a5cbc4b862345791"
server_key_iv: "03cacea5f3909a68c5c30bed038dd225"
server_key_version: "encryptionkey@0"
I20260924 04:03:29.774616 6483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:03:29.777401 6498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:29.778815 6483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:03:29.778937 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "4e2e7e0e64f44abf9ea715aa39cbb843"
format_stamp: "Formatted at 2026-09-24 04:03:29 on dist-test-slave-zcsk"
server_key: "6f5f4e95bf889530a5cbc4b862345791"
server_key_iv: "03cacea5f3909a68c5c30bed038dd225"
server_key_version: "encryptionkey@0"
I20260924 04:03:29.779062 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:29.796579 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:29.799853 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:29.800062 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:29.807152 6483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46295
I20260924 04:03:29.807173 6550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46295 every 8 connection(s)
I20260924 04:03:29.808256 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:29.808784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 6483
I20260924 04:03:29.808916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:29.809192 392 external_mini_cluster.cc:1428] Setting key 457564bf95a2bf1a8fe1ee92481e7dbb
I20260924 04:03:29.811304 6551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:29.817808 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: Bootstrap starting.
Sep 24 04:03:29 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:29.820243 6551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:29.820976 6551 log.cc:824] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:29.822966 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: No bootstrap required, opened a new log
I20260924 04:03:29.823459 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:29.810579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.0.98.62:46295)
0924 04:03:29.810986 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:29.811027 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:29.811065 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:03:29.811727 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:29.811745 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:29.812136 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:29.812421 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:29.813277 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:29.814280 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:29.815363 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:29.815832 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:29.816012 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:29.818611 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:29.818616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:29.818629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:29.818659 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:29.820501 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:29.820970 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:29.820975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:29.820981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:29.821109 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:29.821387 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:29.821391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:29.821393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:29.821818 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:29.822022 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:29.822360 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 04:03:29.822665 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
I20260924 04:03:29.826601 6551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } }
I20260924 04:03:29.826843 6551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:29.826894 6551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2e7e0e64f44abf9ea715aa39cbb843, State: Initialized, Role: FOLLOWER
I20260924 04:03:29.827319 6551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } }
I20260924 04:03:29.827457 6551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:29.827521 6551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:29.827617 6551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:29.828697 6551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } }
I20260924 04:03:29.829056 6551 leader_election.cc:302] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e2e7e0e64f44abf9ea715aa39cbb843; no voters:
I20260924 04:03:29.829370 6551 leader_election.cc:288] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:29.829505 6556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:29.829725 6556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 1 LEADER]: Becoming Leader. State: Replica: 4e2e7e0e64f44abf9ea715aa39cbb843, State: Running, Role: LEADER
I20260924 04:03:29.830027 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } }
I20260924 04:03:29.830567 6551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:29.831595 6558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2e7e0e64f44abf9ea715aa39cbb843. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } }
I20260924 04:03:29.831736 6558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:29.832142 6565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:29.832129 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } }
I20260924 04:03:29.832283 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:29.835319 6565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:29.841303 6565 catalog_manager.cc:1347] Generated new cluster ID: 8689bfcb2c00493cb372365e3f424ece
I20260924 04:03:29.841382 6565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:29.848933 6565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:29.850167 6565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:29.865056 6565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: Generated new TSK 0
I20260924 04:03:29.865978 6565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:03:29.903348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46295
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:30.016232 6577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:30.016624 6577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:30.016692 6577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:30.022127 6577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:30.022218 6577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:30.022305 6577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:30.029443 6577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:30.030764 6577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:30.032053 6577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:30.042004 6585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.042013 6583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.042057 6582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:30.042277 6577 server_base.cc:1034] running on GCE node
I20260924 04:03:30.042698 6577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:30.043388 6577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:30.044579 6577 hybrid_clock.cc:648] HybridClock initialized: now 1790222610044549 us; error 44 us; skew 500 ppm
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:30.047492 6577 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:03:30.048530 6577 webserver.cc:460] Webserver started at http://127.0.98.1:43069/ using document root <none> and password file <none>
I20260924 04:03:30.049110 6577 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:30.049180 6577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:30.049376 6577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:30.051498 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "f7c19b464c064f27bab5b704f1ff6127"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "2369c65502fa1e0474bf7395496527ab"
server_key_iv: "b2e704e66f50c88e19848c289325a59c"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.052091 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f7c19b464c064f27bab5b704f1ff6127"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "2369c65502fa1e0474bf7395496527ab"
server_key_iv: "b2e704e66f50c88e19848c289325a59c"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.055812 6577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:03:30.058300 6592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.059604 6577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:30.059751 6577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "f7c19b464c064f27bab5b704f1ff6127"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "2369c65502fa1e0474bf7395496527ab"
server_key_iv: "b2e704e66f50c88e19848c289325a59c"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.059850 6577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:30.084609 6577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:30.087704 6577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:30.087894 6577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:30.088543 6577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:30.089423 6577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:30.089471 6577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.089526 6577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:30.089555 6577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.098268 6577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46191
I20260924 04:03:30.098308 6705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46191 every 8 connection(s)
I20260924 04:03:30.099152 6577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:30.099522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 6577
I20260924 04:03:30.099664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:30.099973 392 external_mini_cluster.cc:1428] Setting key 0943ec7f28d0342e5e9559bf634f0d81
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:30.115788 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.100946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57799 (local address 127.0.98.62:46295)
0924 04:03:30.101100 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.101104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.102058 (+ 954us) server_negotiation.cc:407] Connection header received
0924 04:03:30.103662 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.103665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.103712 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.103784 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.105018 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.105553 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.106680 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.106910 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.106999 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.112182 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.112185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.112187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.112215 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.113759 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.114281 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.114284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.114286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.114367 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.114687 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.114691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.114692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.114877 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.114962 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.115475 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.115616 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 04:03:30.116642 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.101413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:57799)
0924 04:03:30.101880 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.101933 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.103421 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.104007 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.104015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.104377 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.104868 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.104882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.105676 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.105679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.106567 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.106576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.107060 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.107063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.107220 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.107820 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.107836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.111953 (+ 4117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.113882 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.113889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.113908 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.114184 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.114448 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.114451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.114454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.114603 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.114971 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.114986 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.115245 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.115419 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.115766 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260924 04:03:30.118083 6706 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46295
I20260924 04:03:30.118323 6706 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:30.118811 6706 heartbeater.cc:511] Master 127.0.98.62:46295 requested a full tablet report, sending...
I20260924 04:03:30.120373 6515 ts_manager.cc:194] Registered new tserver with Master: f7c19b464c064f27bab5b704f1ff6127 (127.0.98.1:46191)
I20260924 04:03:30.121503 6515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:03:30.127746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46295
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:30.241114 6711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:30.241430 6711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:30.241475 6711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:30.245560 6711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:30.245636 6711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:30.245699 6711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:30.251286 6711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:30.252431 6711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:30.253504 6711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:30.263615 6711 server_base.cc:1034] running on GCE node
W20260924 04:03:30.263561 6719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.263633 6717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.263561 6716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:30.265185 6711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:30.265883 6711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:30.267052 6711 hybrid_clock.cc:648] HybridClock initialized: now 1790222610267030 us; error 39 us; skew 500 ppm
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:30.269935 6711 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:03:30.270937 6711 webserver.cc:460] Webserver started at http://127.0.98.2:40259/ using document root <none> and password file <none>
I20260924 04:03:30.271526 6711 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:30.271595 6711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:30.271791 6711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:30.274001 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "7ddabd7fc4244e1aaee11067659f6d73"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "c51d8b52304a9d59d9a6c1fbc894326f"
server_key_iv: "1590c74a91c09e36c22569998d86214e"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.274583 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7ddabd7fc4244e1aaee11067659f6d73"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "c51d8b52304a9d59d9a6c1fbc894326f"
server_key_iv: "1590c74a91c09e36c22569998d86214e"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.278609 6711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:03:30.281279 6726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.282719 6711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:03:30.282830 6711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "7ddabd7fc4244e1aaee11067659f6d73"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "c51d8b52304a9d59d9a6c1fbc894326f"
server_key_iv: "1590c74a91c09e36c22569998d86214e"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.282950 6711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:30.297170 6711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:30.300479 6711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:30.300675 6711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:30.301303 6711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:30.302157 6711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:30.302206 6711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.302248 6711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:30.302276 6711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.311098 6711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45117
I20260924 04:03:30.311118 6839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45117 every 8 connection(s)
I20260924 04:03:30.312050 6711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:30.314222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 6711
I20260924 04:03:30.314337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:30.314577 392 external_mini_cluster.cc:1428] Setting key ef37a1781a60b773f38cebd1e2be1845
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 04:03:30.325315 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.313842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44591 (local address 127.0.98.62:46295)
0924 04:03:30.313980 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.314002 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.314848 (+ 846us) server_negotiation.cc:407] Connection header received
0924 04:03:30.316331 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.316335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.316383 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.316473 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.317625 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.318134 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.319243 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.319461 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.319548 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.322022 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.322025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.322027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.322051 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.323383 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.323861 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.323865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.323866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.323976 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.324268 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.324270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.324271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.324442 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.324512 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.325024 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.325153 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:03:30.326161 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.314221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:44591)
0924 04:03:30.314690 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.314741 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.316127 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.316613 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.316620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.316968 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.317484 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.317497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.318252 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.318256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.319131 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.319140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.319597 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.319601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.319787 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.320414 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.320431 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.321856 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.323492 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.323498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.323516 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.323768 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.324063 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.324065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.324067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.324200 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.324513 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.324517 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.324798 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.324957 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.325284 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":134,"threads_started":1}
I20260924 04:03:30.327378 6840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46295
I20260924 04:03:30.327625 6840 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:30.328150 6840 heartbeater.cc:511] Master 127.0.98.62:46295 requested a full tablet report, sending...
I20260924 04:03:30.329205 6515 ts_manager.cc:194] Registered new tserver with Master: 7ddabd7fc4244e1aaee11067659f6d73 (127.0.98.2:45117)
I20260924 04:03:30.329751 6515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:03:30.343091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46295
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:30.455158 6845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:30.455441 6845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:30.455483 6845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:30.459692 6845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:30.459775 6845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:30.459856 6845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:30.465307 6845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:30.466370 6845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:30.467348 6845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:30.476845 6853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.476845 6851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.476989 6850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:30.477102 6845 server_base.cc:1034] running on GCE node
I20260924 04:03:30.477715 6845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:30.478358 6845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:30.479529 6845 hybrid_clock.cc:648] HybridClock initialized: now 1790222610479494 us; error 38 us; skew 500 ppm
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:30.482237 6845 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:03:30.483237 6845 webserver.cc:460] Webserver started at http://127.0.98.3:44107/ using document root <none> and password file <none>
I20260924 04:03:30.483837 6845 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:30.483932 6845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:30.484153 6845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:30.486352 6845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "4adba512c0ac497b88071971d6c71848"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "09463fc22f3280352f30a6578896459e"
server_key_iv: "3483c753742677ae2dab904deeaad56b"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.486893 6845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4adba512c0ac497b88071971d6c71848"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "09463fc22f3280352f30a6578896459e"
server_key_iv: "3483c753742677ae2dab904deeaad56b"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.490785 6845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 04:03:30.493320 6860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.494590 6845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:30.494711 6845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "4adba512c0ac497b88071971d6c71848"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "09463fc22f3280352f30a6578896459e"
server_key_iv: "3483c753742677ae2dab904deeaad56b"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.494810 6845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:30.511458 6845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:30.514395 6845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:30.514570 6845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:30.515139 6845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:30.515969 6845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:30.516021 6845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.516075 6845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:30.516100 6845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.524333 6845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41799
I20260924 04:03:30.524353 6973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41799 every 8 connection(s)
I20260924 04:03:30.525213 6845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:30.529369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 6845
I20260924 04:03:30.529484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:30.529775 392 external_mini_cluster.cc:1428] Setting key 236c15e80518aa1f051a8c7da2bc6fb4
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:30.538245 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.526912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49111 (local address 127.0.98.62:46295)
0924 04:03:30.527072 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.527076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.527925 (+ 849us) server_negotiation.cc:407] Connection header received
0924 04:03:30.529252 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.529255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.529300 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.529372 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.530757 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.531401 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.532723 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.532936 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.533045 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.535270 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.535273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.535275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.535296 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.536429 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.536877 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.536880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.536882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.536949 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.537246 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.537249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.537251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.537415 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.537493 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.537964 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.538086 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:03:30.538993 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.527323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:49111)
0924 04:03:30.527761 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.527803 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.529028 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.529596 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.529606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.530071 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.530623 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.530640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.531516 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.531518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.532608 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.532619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.533045 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.533048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.533207 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.533743 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.533760 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.535116 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.536535 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.536540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.536556 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.536788 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.537031 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.537035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.537037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.537163 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.537514 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.537521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.537750 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.537911 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.538218 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260924 04:03:30.540025 6974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46295
I20260924 04:03:30.540241 6974 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:30.540704 6974 heartbeater.cc:511] Master 127.0.98.62:46295 requested a full tablet report, sending...
I20260924 04:03:30.541648 6515 ts_manager.cc:194] Registered new tserver with Master: 4adba512c0ac497b88071971d6c71848 (127.0.98.3:41799)
I20260924 04:03:30.542176 6515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49111
I20260924 04:03:30.543507 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 04:03:30.568455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43963
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46295,127.0.98.61:43963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:30.689215 6979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:30.689497 6979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:30.689543 6979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:30.693811 6979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:30.693876 6979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:30.693893 6979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:30.693909 6979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:30.693931 6979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:30.699599 6979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46295,127.0.98.61:43963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
W20260924 04:03:30.699839 6979 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:46295,127.0.98.61:43963'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 04:03:30.700775 6979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:30.701798 6979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:30.710060 6987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.710088 6984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.710045 6985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:30.710419 6979 server_base.cc:1034] running on GCE node
I20260924 04:03:30.710970 6979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:30.712167 6979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:30.713325 6979 hybrid_clock.cc:648] HybridClock initialized: now 1790222610713303 us; error 33 us; skew 500 ppm
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:30.715927 6979 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:03:30.716830 6979 webserver.cc:460] Webserver started at http://127.0.98.61:43549/ using document root <none> and password file <none>
I20260924 04:03:30.717370 6979 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:30.717433 6979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:30.717634 6979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:30.719682 6979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data/instance:
uuid: "0b7890dab2c943f9bb8610647603ee18"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "cc87c7936a85ef2ca40c12517a4f16b4"
server_key_iv: "15cc0d7a317224f3ea57270a27ee8db2"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.720359 6979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal/instance:
uuid: "0b7890dab2c943f9bb8610647603ee18"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "cc87c7936a85ef2ca40c12517a4f16b4"
server_key_iv: "15cc0d7a317224f3ea57270a27ee8db2"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.724169 6979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:03:30.726744 6994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.728152 6979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:30.728281 6979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
uuid: "0b7890dab2c943f9bb8610647603ee18"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "cc87c7936a85ef2ca40c12517a4f16b4"
server_key_iv: "15cc0d7a317224f3ea57270a27ee8db2"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.728391 6979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:30.746757 6979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:30.749785 6979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:30.749984 6979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:30.756824 6979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43963
I20260924 04:03:30.757069 7046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43963 every 8 connection(s)
I20260924 04:03:30.757820 6979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:03:30.760644 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:30.764381 7047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:03:30.764935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 6979
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:03:30.776440 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.766178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.0.98.62:46295)
0924 04:03:30.766322 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.766326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.766872 (+ 546us) server_negotiation.cc:407] Connection header received
0924 04:03:30.767613 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.767616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.767667 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.767734 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.768612 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.769122 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.770080 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.770285 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.770368 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.772716 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.772719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.772721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.772744 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.774112 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.774613 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.774617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.774618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.774693 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.774984 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.774988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.774989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.775197 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.775281 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.775769 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.775893 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:03:30.777227 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.766290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.0.98.61:43963)
0924 04:03:30.766672 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.766688 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.766727 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:03:30.767432 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.767450 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.767646 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.767811 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.768619 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.769446 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.770232 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.770576 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.770662 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.773048 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.773055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.773067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.773086 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.774560 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.774942 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.774947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.774948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.775015 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.775249 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.775252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.775254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.775577 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.775746 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.775983 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.776166 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":263,"spinlock_wait_cycles":5248,"thread_start_us":107,"threads_started":1}
I20260924 04:03:30.777227 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.766290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60100)
0924 04:03:30.766753 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.766789 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.767437 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.767802 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.767808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.768219 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.768484 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.768497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.769235 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.769238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.769965 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.769982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.770412 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.770416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.770578 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.771101 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.771121 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.772533 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.774208 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.774216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.774226 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.774527 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.774770 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.774773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.774775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.774903 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.775294 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.775301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.775568 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.775715 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.776275 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":131,"spinlock_wait_cycles":28800,"thread_start_us":160,"threads_started":1}
I20260924 04:03:30.778555 7047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:03:30.785393 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.779030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.0.1:34744)
0924 04:03:30.779133 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.779147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.779217 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.779408 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.779412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.779589 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.779680 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.779685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.780380 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.780383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.780890 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.780896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.781274 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.781277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.781340 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.781709 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.781719 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.782931 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.784324 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.784326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.784328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.784500 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.784699 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.784701 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.784702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.784762 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.785023 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.785025 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.785089 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.785156 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.785253 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 04:03:30.785418 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.779127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.0.98.61:43963)
0924 04:03:30.779220 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.779224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.779237 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:30.779276 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.779280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.779319 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.779384 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.779767 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.780302 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.780979 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.781198 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.781272 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.783030 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.783033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.783035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.783058 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.784240 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.784577 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.784580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.784581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.784642 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.784816 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.784818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.784819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.784964 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.785027 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.785193 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.785300 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":37}
I20260924 04:03:30.788606 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Bootstrap starting.
I20260924 04:03:30.790872 7047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:30.791580 7047 log.cc:824] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:30.793423 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: No bootstrap required, opened a new log
I20260924 04:03:30.796370 7047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } }
I20260924 04:03:30.796587 7047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:30.796643 7047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7890dab2c943f9bb8610647603ee18, State: Initialized, Role: FOLLOWER
I20260924 04:03:30.797053 7047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } }
I20260924 04:03:30.797469 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } } }
I20260924 04:03:30.797595 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:03:30.798432 7047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:03:30.802935 7064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:03:30.803030 7064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:03:30.804801 6979 master_runner.cc:428] Detected that this master 0b7890dab2c943f9bb8610647603ee18 is joining an existing cluster
I20260924 04:03:30.804857 6979 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43963
I20260924 04:03:30.815361 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.806974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.0.98.61:43963)
0924 04:03:30.807502 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.807506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.807694 (+ 188us) server_negotiation.cc:407] Connection header received
0924 04:03:30.807821 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.807824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.807856 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.807899 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.808495 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.809241 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.810341 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.810654 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.810722 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.812074 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.812078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.812080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.812102 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.813778 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.814230 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.814234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.814236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.814306 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.814536 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.814539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.814540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.814734 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.814798 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.815089 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.815202 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469}
I20260924 04:03:30.815706 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.806847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.0.98.62:46295)
0924 04:03:30.807095 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.807099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.807321 (+ 222us) server_negotiation.cc:407] Connection header received
0924 04:03:30.807456 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.807459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.807511 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.807581 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.808231 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.808971 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.810184 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.810526 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.810615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.812449 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.812453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.812456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.812483 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.813907 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.814371 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.814374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.814376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.814464 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.814706 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.814710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.814711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.814896 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.814967 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.815453 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.815561 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260924 04:03:30.816046 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.806811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60114)
0924 04:03:30.807223 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.807238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.807344 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.807697 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.807700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.807966 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.808107 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.808114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.809095 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.809098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.810059 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.810068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.810748 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.810752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.810840 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.811817 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.811831 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.812290 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.814024 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.814027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.814029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.814277 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.814548 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.814551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.814553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.814631 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.815002 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.815005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.815085 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.815163 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.815294 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260924 04:03:30.816438 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.806885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.0.1:34754)
0924 04:03:30.807626 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.807638 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.807741 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.808051 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.808054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.808258 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.808384 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.808391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.809357 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.809360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.810225 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.810235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.810853 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.810856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.810939 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.811469 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.811484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.811939 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.813884 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.813887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.813889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.814141 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.814389 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.814392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.814394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.814461 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.814850 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.814853 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.814936 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.816160 (+ 1224us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.816286 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":92,"threads_started":1}
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:03:30.833088 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.820940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.0.98.61:43963)
0924 04:03:30.821067 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.821072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.823700 (+ 2628us) server_negotiation.cc:407] Connection header received
0924 04:03:30.823882 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.823885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.823977 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.824078 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.824686 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.825462 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.826373 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.826736 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.826836 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.829304 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.829308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.829310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.829334 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.830972 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.831487 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.831490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.831492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.831584 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.831922 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.831926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.831928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.832159 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.832267 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.832758 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.832888 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:30.833148 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.820913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.0.1:34768)
0924 04:03:30.823567 (+ 2654us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.823607 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.823745 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.824107 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.824112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.824474 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.824582 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.824587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.825589 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.825592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.826268 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.826277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.826853 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.826856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.826943 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.827523 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.827535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.829139 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.831081 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.831093 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.831100 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.831396 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.831692 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.831696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.831698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.831794 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.832326 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.832334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.832509 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.832672 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.832950 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2487,"thread_start_us":101,"threads_started":1}
I20260924 04:03:30.834707 6515 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43963: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0b7890dab2c943f9bb8610647603ee18"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43963
}
attrs {
promote: true
}
}
dest_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843"
cas_config_opid_index: -1
I20260924 04:03:30.835433 6515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } }
I20260924 04:03:30.837646 7075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0b7890dab2c943f9bb8610647603ee18 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } }
I20260924 04:03:30.838344 7021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 04:03:30.839395 7075 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43963
I20260924 04:03:30.839524 7021 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Refusing update from remote peer 4e2e7e0e64f44abf9ea715aa39cbb843: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 04:03:30.839938 6979 master.cc:537] Master@127.0.98.61:43963 shutting down...
I20260924 04:03:30.840052 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } }
I20260924 04:03:30.840157 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:30.840330 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
W20260924 04:03:30.843334 6500 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 04:03:30.845285 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 -> Peer 0b7890dab2c943f9bb8610647603ee18 (127.0.98.61:43963): Couldn't send request to peer 0b7890dab2c943f9bb8610647603ee18. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:03:30.845882 6979 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:30.846213 6979 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:30.846321 6979 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: stopping tablet replica
I20260924 04:03:30.851236 6979 master.cc:552] Master@127.0.98.61:43963 shutdown complete.
I20260924 04:03:30.851295 6979 master_runner.cc:305] Clearing existing system tablet
I20260924 04:03:30.851526 6979 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:30.851581 6979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:30.852919 6979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:30.853735 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.854031 6979 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:30.854166 6979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:30.854208 6979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
uuid: "0b7890dab2c943f9bb8610647603ee18"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "cc87c7936a85ef2ca40c12517a4f16b4"
server_key_iv: "15cc0d7a317224f3ea57270a27ee8db2"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.856446 6979 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 04:03:30.859313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37355
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46295,127.0.98.61:43963,127.0.98.60:37355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:03:30.859319 6979 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:03:30.859423 6979 log.cc:1196] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 04:03:30.859678 6979 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Deleting consensus metadata
I20260924 04:03:30.859838 6979 master_runner.cc:315] Copying system tablet from 127.0.98.62:46295
I20260924 04:03:30.861083 6979 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46295
I20260924 04:03:30.869449 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.862162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.98.62:46295)
0924 04:03:30.862300 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:30.862304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:30.862412 (+ 108us) server_negotiation.cc:407] Connection header received
0924 04:03:30.862533 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:30.862535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:30.862579 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:30.862650 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:30.863196 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.863774 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.864873 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.865126 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.865212 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.866428 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:30.866432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:30.866434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:30.866458 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:30.867713 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.868196 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.868200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.868202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.868293 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:30.868564 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:30.868567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:30.868568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:30.868785 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:30.868877 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:30.869167 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 04:03:30.869282 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:03:30.869470 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:30.862003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60130)
0924 04:03:30.862317 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:30.862332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:30.862451 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:30.862681 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:30.862684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:30.862890 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:30.863015 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.863021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.863875 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.863879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.864757 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:30.864768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:30.865239 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:30.865243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:30.865340 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:30.865818 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:30.865832 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:30.866276 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:30.867816 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.867820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.867822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.868104 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.868382 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:30.868385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:30.868386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:30.868479 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:30.868878 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:30.868881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:30.868991 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 04:03:30.869113 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 04:03:30.869267 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260924 04:03:30.870324 6535 tablet_copy_service.cc:140] P 4e2e7e0e64f44abf9ea715aa39cbb843: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0b7890dab2c943f9bb8610647603ee18 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60130)
I20260924 04:03:30.870540 6535 tablet_copy_service.cc:161] P 4e2e7e0e64f44abf9ea715aa39cbb843: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0b7890dab2c943f9bb8610647603ee18 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60130: session id = 0b7890dab2c943f9bb8610647603ee18-00000000000000000000000000000000
I20260924 04:03:30.872539 6535 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:03:30.874300 6979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:03:30.876675 6979 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: tablet copy: Starting download of 0 data blocks...
I20260924 04:03:30.876869 6979 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: tablet copy: Starting download of 1 WAL segments...
I20260924 04:03:30.878805 6979 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:03:30.880831 6535 tablet_copy_service.cc:342] P 4e2e7e0e64f44abf9ea715aa39cbb843: Request end of tablet copy session 0b7890dab2c943f9bb8610647603ee18-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60130
I20260924 04:03:30.880988 6535 tablet_copy_service.cc:433] P 4e2e7e0e64f44abf9ea715aa39cbb843: ending tablet copy session 0b7890dab2c943f9bb8610647603ee18-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0b7890dab2c943f9bb8610647603ee18
I20260924 04:03:30.889526 6979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:30.891279 7088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:30.893174 6979 server_base.cc:1034] running on GCE node
W20260924 04:03:30.893118 7091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:30.896375 7089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:30.896631 6979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:30.896950 6979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:30.898149 6979 hybrid_clock.cc:648] HybridClock initialized: now 1790222610898128 us; error 51 us; skew 500 ppm
Sep 24 04:03:30 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:30.899760 6979 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:03:30.900297 6979 webserver.cc:460] Webserver started at http://127.0.98.61:44831/ using document root <none> and password file <none>
I20260924 04:03:30.900488 6979 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:30.900532 6979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:30.901867 6979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:30.902875 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:30.903136 6979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:30.903245 6979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
uuid: "0b7890dab2c943f9bb8610647603ee18"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "cc87c7936a85ef2ca40c12517a4f16b4"
server_key_iv: "15cc0d7a317224f3ea57270a27ee8db2"
server_key_version: "encryptionkey@0"
I20260924 04:03:30.903388 6979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:30.939698 6979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:30.942142 6979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:30.947319 6979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43963
I20260924 04:03:30.947355 7150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43963 every 8 connection(s)
I20260924 04:03:30.948129 6979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:03:30.950028 7151 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:03:30.951138 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Bootstrap starting.
I20260924 04:03:30.962111 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:03:30.962455 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: Bootstrap complete.
I20260924 04:03:30.963052 7151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } }
I20260924 04:03:30.963188 7151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b7890dab2c943f9bb8610647603ee18, State: Initialized, Role: LEARNER
I20260924 04:03:30.963297 7151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } }
I20260924 04:03:30.963765 7151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:30.963742 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } }
I20260924 04:03:30.964062 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:03:30.967428 7165 catalog_manager.cc:1259] Loaded cluster ID: 8689bfcb2c00493cb372365e3f424ece
I20260924 04:03:30.967496 7165 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: loading cluster ID for follower catalog manager: success
I20260924 04:03:30.969697 7165 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: acquiring CA information for follower catalog manager: success
I20260924 04:03:30.971444 7165 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 04:03:30.990617 7083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:30.990942 7083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:30.991011 7083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:30.995397 7083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:30.995486 7083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:30.995512 7083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:30.995534 7083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:30.995554 7083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:31.001410 7083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46295,127.0.98.61:43963,127.0.98.60:37355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:31.002569 7083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:31.003680 7083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:31.012220 7170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.012292 7171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.012205 7173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:31.012571 7083 server_base.cc:1034] running on GCE node
I20260924 04:03:31.013269 7083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:31.014420 7083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:31.015602 7083 hybrid_clock.cc:648] HybridClock initialized: now 1790222611015572 us; error 50 us; skew 500 ppm
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:31.018442 7083 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:03:31.019384 7083 webserver.cc:460] Webserver started at http://127.0.98.60:35727/ using document root <none> and password file <none>
I20260924 04:03:31.019987 7083 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:31.020087 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:31.020289 7083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:31.022364 7083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data/instance:
uuid: "fc33482acaf44742a3028d36fbfd81ee"
format_stamp: "Formatted at 2026-09-24 04:03:31 on dist-test-slave-zcsk"
server_key: "3a9dcbcb144484f37b0c7f3a4f0ebb1a"
server_key_iv: "79993da0dd17bf15c4844954c0d8afe9"
server_key_version: "encryptionkey@0"
I20260924 04:03:31.022909 7083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal/instance:
uuid: "fc33482acaf44742a3028d36fbfd81ee"
format_stamp: "Formatted at 2026-09-24 04:03:31 on dist-test-slave-zcsk"
server_key: "3a9dcbcb144484f37b0c7f3a4f0ebb1a"
server_key_iv: "79993da0dd17bf15c4844954c0d8afe9"
server_key_version: "encryptionkey@0"
I20260924 04:03:31.026839 7083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:03:31.029585 7180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.030980 7083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:03:31.031123 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
uuid: "fc33482acaf44742a3028d36fbfd81ee"
format_stamp: "Formatted at 2026-09-24 04:03:31 on dist-test-slave-zcsk"
server_key: "3a9dcbcb144484f37b0c7f3a4f0ebb1a"
server_key_iv: "79993da0dd17bf15c4844954c0d8afe9"
server_key_version: "encryptionkey@0"
I20260924 04:03:31.031245 7083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:31.050876 7083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:31.054235 7083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:31.054423 7083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:31.061636 7083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37355
I20260924 04:03:31.061935 7232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37355 every 8 connection(s)
I20260924 04:03:31.062603 7083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:03:31.065748 7233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:31.068588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 7083
I20260924 04:03:31.069953 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:03:31.082255 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.071820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.0.98.62:46295)
0924 04:03:31.072015 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.072020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.072186 (+ 166us) server_negotiation.cc:407] Connection header received
0924 04:03:31.072316 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.072318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.072370 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.072442 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.073302 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.073829 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.075037 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.075263 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.075364 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.077998 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.078003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.078005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.078042 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.079797 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.080488 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.080492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.080494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.080592 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.081042 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.081046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.081047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.081271 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.081343 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.081927 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.082087 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 04:03:31.083256 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.070104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.0.98.60:37355)
0924 04:03:31.070560 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.070576 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.070612 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:03:31.071259 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.071277 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.071679 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.072585 (+ 906us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.072925 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.074384 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.075390 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.075797 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.075885 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.078458 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.078478 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.078494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.078523 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.080381 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.080840 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.080846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.080848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.080921 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.081175 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.081178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.081179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.081693 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.081876 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.082079 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.082263 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":306,"thread_start_us":118,"threads_started":1}
I20260924 04:03:31.083890 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.071619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60140)
0924 04:03:31.072055 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.072108 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.072227 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.072520 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.072530 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.072964 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.073161 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.073171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.073949 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.073952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.074888 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.074898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.075413 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.075416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.075574 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.076187 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.076205 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.077780 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.079996 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.080004 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.080017 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.080379 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.080688 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.080691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.080693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.080863 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.081364 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.081370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.081694 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.081864 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.083665 (+ 1801us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1476,"spinlock_wait_cycles":7040,"thread_start_us":129,"threads_started":1}
I20260924 04:03:31.085106 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:03:31.093688 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.085589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.0.1:34770)
0924 04:03:31.085737 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.085752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.085907 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.086286 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.086289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.086447 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.086559 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.086565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.087450 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.087454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.088062 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.088071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.088532 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.088534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.088624 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.089146 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.089157 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.090374 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.092237 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.092240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.092242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.092447 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.092765 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.092768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.092769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.092864 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.093223 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.093225 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.093315 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.093433 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.093558 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 04:03:31.093789 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.085729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.0.98.61:43963)
0924 04:03:31.086025 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.086030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.086044 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:31.086086 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.086089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.086136 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.086228 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.086659 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.087324 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.088164 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.088401 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.088495 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.090540 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.090543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.090546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.090570 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.092119 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.092552 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.092556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.092557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.092661 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.092938 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.092940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.092941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.093123 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.093195 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.093524 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.093649 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":117,"threads_started":1}
I20260924 04:03:31.094461 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:03:31.102219 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.094973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.0.1:34450)
0924 04:03:31.095111 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.095126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.095199 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.095385 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.095388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.095538 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.095614 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.095622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.096381 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.096384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.096871 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.096879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.097275 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.097278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.097355 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.097791 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.097803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.099039 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.100820 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.100823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.100824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.101003 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.101288 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.101290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.101291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.101372 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.101756 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.101759 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.101835 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.101914 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.102095 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":2}
I20260924 04:03:31.102331 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.095081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.0.98.60:37355)
0924 04:03:31.095167 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.095171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.095182 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:03:31.095265 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.095268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.095302 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.095358 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.095712 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.096270 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.096968 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.097186 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.097266 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.099199 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.099202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.099204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.099226 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.100710 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.101094 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.101098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.101100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.101184 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.101454 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.101457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.101459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.101665 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.101758 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.101968 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.102160 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":27}
I20260924 04:03:31.105634 7233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Bootstrap starting.
I20260924 04:03:31.107973 7233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:31.108670 7233 log.cc:824] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:31.110658 7233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: No bootstrap required, opened a new log
I20260924 04:03:31.113715 7233 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } }
I20260924 04:03:31.113940 7233 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:31.113984 7233 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc33482acaf44742a3028d36fbfd81ee, State: Initialized, Role: FOLLOWER
I20260924 04:03:31.114380 7233 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } }
I20260924 04:03:31.114811 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } } }
I20260924 04:03:31.114943 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:03:31.115533 7233 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:03:31.120213 7250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:03:31.120309 7250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:03:31.123445 7083 master_runner.cc:428] Detected that this master fc33482acaf44742a3028d36fbfd81ee is joining an existing cluster
I20260924 04:03:31.123510 7083 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:37355
I20260924 04:03:31.124006 6706 heartbeater.cc:503] Master 127.0.98.62:46295 was elected leader, sending a full tablet report...
I20260924 04:03:31.134518 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.126368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.98.62:46295)
0924 04:03:31.126481 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.126485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.126498 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:31.126538 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.126541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.126587 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.126672 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.127222 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.128001 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.129123 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.129461 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.129530 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.131007 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.131011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.131014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.131040 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.132820 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.133279 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.133283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.133285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.133369 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.133666 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.133669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.133671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.133848 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.133912 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.134260 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.134358 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:03:31.135071 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.125924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60152)
0924 04:03:31.126346 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.126360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.126445 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.126702 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.126705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.126944 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.127102 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.127109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.128123 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.128126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.129002 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.129010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.129691 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.129694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.129792 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.130321 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.130335 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.130843 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.132938 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.132941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.132943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.133167 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.133482 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.133485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.133487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.133569 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.133967 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.133969 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.134082 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.134732 (+ 650us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.134873 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260924 04:03:31.135262 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.126051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.0.1:34772)
0924 04:03:31.126896 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.126908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.127033 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.127362 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.127364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.127621 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.127724 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.127731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.128759 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.128762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.129745 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.129752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.130681 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.130684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.131028 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.131517 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.131531 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.131925 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.133686 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.133690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.133692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.133932 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.134197 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.134200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.134202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.134326 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.134826 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.134829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.134909 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.135008 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.135121 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":97,"thread_start_us":108,"threads_started":1}
I20260924 04:03:31.135403 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.125991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.0.98.61:43963)
0924 04:03:31.126113 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.126118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.126978 (+ 860us) server_negotiation.cc:407] Connection header received
0924 04:03:31.127135 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.127138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.127190 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.127239 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.127848 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.128636 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.129856 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.130462 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.130539 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.132044 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.132048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.132051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.132079 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.133578 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.134031 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.134035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.134037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.134117 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.134471 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.134474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.134476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.134701 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.134788 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.135086 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.135214 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:03:31.136912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 6577
I20260924 04:03:31.142912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46191
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46295
--webserver_port=43069
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963,127.0.98.60:37355 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:03:31.147431 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.126605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.0.1:34454)
0924 04:03:31.126977 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.126990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.127101 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.127384 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.127386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.127683 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.127797 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.127804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.128804 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.128807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.129829 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.129839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.130464 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.130467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.130549 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.130974 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.130987 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.144257 (+ 13270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.146057 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.146060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.146063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.146271 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.146514 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.146516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.146519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.146587 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.146964 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.146966 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.147035 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.147112 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.147237 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":75,"thread_start_us":84,"threads_started":1}
I20260924 04:03:31.148003 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.126451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.0.98.60:37355)
0924 04:03:31.126545 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.126548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.127050 (+ 502us) server_negotiation.cc:407] Connection header received
0924 04:03:31.127185 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.127187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.127221 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.127262 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.127935 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.128672 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.129943 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.130255 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.130324 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.144396 (+ 14072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.144400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.144402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.144426 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.145864 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.146362 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.146365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.146366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.146434 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.146665 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.146668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.146669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.146846 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.146936 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.147727 (+ 791us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.147830 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:31.151018 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.140687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.0.98.60:37355)
0924 04:03:31.140990 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.140997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.141011 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:31.141081 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.141084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.141136 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.141223 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.141689 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.142531 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.143553 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.143957 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.145691 (+ 1734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.146712 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.146716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.146719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.146744 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.149158 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.149682 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.149686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.149687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.149778 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.150111 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.150115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.150117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.150326 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.150428 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.150724 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.150846 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":121,"threads_started":1}
I20260924 04:03:31.151027 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.140219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.0.1:34466)
0924 04:03:31.140471 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.140485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.140561 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.141267 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.141270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.141472 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.141583 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.141590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.142646 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.142650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.143435 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.143447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.144091 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.144094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.144199 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.144866 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.144882 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.146572 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.149303 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.149307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.149310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.149565 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.149892 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.149896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.149898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.150003 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.150431 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.150435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.150548 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.150669 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.150829 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260924 04:03:31.152705 6515 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:37355: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 37355
}
attrs {
promote: true
}
}
dest_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843"
cas_config_opid_index: 5
I20260924 04:03:31.153409 6515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } }
I20260924 04:03:31.154569 7074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc33482acaf44742a3028d36fbfd81ee (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.155828 7207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:31.156076 7074 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:37355
I20260924 04:03:31.156237 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.156322 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:31.157263 7207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Refusing update from remote peer 4e2e7e0e64f44abf9ea715aa39cbb843: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:03:31.157686 7083 master.cc:537] Master@127.0.98.60:37355 shutting down...
I20260924 04:03:31.157905 7075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:03:31.159868 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 -> Peer fc33482acaf44742a3028d36fbfd81ee (127.0.98.60:37355): Couldn't send request to peer fc33482acaf44742a3028d36fbfd81ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:03:31.162513 7083 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:31.162846 7083 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:31.162973 7083 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: stopping tablet replica
I20260924 04:03:31.166998 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.154734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.0.1:34786)
0924 04:03:31.161943 (+ 7209us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.161962 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.162118 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.162563 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.162566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.162666 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:31.162772 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.162779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.164018 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.164022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.165377 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.165388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.166527 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.166531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.166630 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.166682 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.166750 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.166839 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7116}
I20260924 04:03:31.167392 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.155030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.0.98.61:43963)
0924 04:03:31.155141 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.155145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.162034 (+ 6889us) server_negotiation.cc:407] Connection header received
0924 04:03:31.162243 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.162246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.162312 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.162409 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:31.162902 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.163753 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.165510 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.166193 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.166302 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.166464 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.167201 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.167281 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:31.167893 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 LEARNER]: Refusing update from remote peer 4e2e7e0e64f44abf9ea715aa39cbb843: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:03:31.168536 7075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 04:03:31.170629 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc33482acaf44742a3028d36fbfd81ee (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.172605 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2e7e0e64f44abf9ea715aa39cbb843. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.172719 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:03:31.172791 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: true } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.172852 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:03:31.173197 7075 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: attempting to promote NON_VOTER 0b7890dab2c943f9bb8610647603ee18 to VOTER
I20260924 04:03:31.173678 7075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } }
I20260924 04:03:31.175082 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.174678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.0.1:34468)
0924 04:03:31.174927 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.174998 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260924 04:03:31.175336 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 LEARNER]: Refusing update from remote peer 4e2e7e0e64f44abf9ea715aa39cbb843: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 04:03:31.175873 7263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 04:03:31.176515 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.176184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.0.1:34476)
0924 04:03:31.176399 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.176448 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260924 04:03:31.177213 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 -> Peer fc33482acaf44742a3028d36fbfd81ee (127.0.98.60:37355): Couldn't send request to peer fc33482acaf44742a3028d36fbfd81ee. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:03:31.177837 7074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0b7890dab2c943f9bb8610647603ee18 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.178275 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0b7890dab2c943f9bb8610647603ee18 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.179461 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.179575 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:31.179697 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.179759 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:31.180608 7262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.180730 7262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:03:31.181054 7083 master.cc:552] Master@127.0.98.60:37355 shutdown complete.
I20260924 04:03:31.181097 7083 master_runner.cc:305] Clearing existing system tablet
I20260924 04:03:31.181313 7083 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:31.181353 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:31.182598 7083 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 04:03:31.183589 7266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.183771 7083 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:31.183878 7083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:31.183941 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
uuid: "fc33482acaf44742a3028d36fbfd81ee"
format_stamp: "Formatted at 2026-09-24 04:03:31 on dist-test-slave-zcsk"
server_key: "3a9dcbcb144484f37b0c7f3a4f0ebb1a"
server_key_iv: "79993da0dd17bf15c4844954c0d8afe9"
server_key_version: "encryptionkey@0"
I20260924 04:03:31.186039 7083 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:31.188467 7083 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:03:31.188562 7083 log.cc:1196] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 04:03:31.188793 7083 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Deleting consensus metadata
I20260924 04:03:31.188951 7083 master_runner.cc:315] Copying system tablet from 127.0.98.62:46295
I20260924 04:03:31.190090 7083 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46295
I20260924 04:03:31.200065 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.191579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60162)
0924 04:03:31.191856 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.191873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.192013 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.192265 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.192267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.192507 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.192637 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.192644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.193682 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.193686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.194558 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.194569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.195164 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.195168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.195262 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.195859 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.195878 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.196393 (+ 515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.198330 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.198333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.198335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.198593 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.198889 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.198892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.198894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.198974 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.199436 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.199439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.199553 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.199658 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.199812 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260924 04:03:31.200080 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.191870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.0.98.62:46295)
0924 04:03:31.192024 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.192028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.192045 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:03:31.192095 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.192098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.192145 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.192228 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.192761 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.193545 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.194689 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.195039 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.195147 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.196543 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.196547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.196550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.196581 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.198206 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.198698 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.198701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.198703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.198785 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.199077 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.199081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.199083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.199342 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.199440 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.199739 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.199882 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 04:03:31.201009 6535 tablet_copy_service.cc:140] P 4e2e7e0e64f44abf9ea715aa39cbb843: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fc33482acaf44742a3028d36fbfd81ee ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60162)
I20260924 04:03:31.201114 6535 tablet_copy_service.cc:161] P 4e2e7e0e64f44abf9ea715aa39cbb843: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fc33482acaf44742a3028d36fbfd81ee at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60162: session id = fc33482acaf44742a3028d36fbfd81ee-00000000000000000000000000000000
I20260924 04:03:31.202487 6535 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:03:31.203964 7083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:03:31.206349 7083 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: tablet copy: Starting download of 0 data blocks...
I20260924 04:03:31.206529 7083 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: tablet copy: Starting download of 1 WAL segments...
I20260924 04:03:31.208153 7083 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:03:31.210137 6535 tablet_copy_service.cc:342] P 4e2e7e0e64f44abf9ea715aa39cbb843: Request end of tablet copy session fc33482acaf44742a3028d36fbfd81ee-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60162
I20260924 04:03:31.210220 6535 tablet_copy_service.cc:433] P 4e2e7e0e64f44abf9ea715aa39cbb843: ending tablet copy session fc33482acaf44742a3028d36fbfd81ee-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fc33482acaf44742a3028d36fbfd81ee
I20260924 04:03:31.218146 7083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:31.220132 7273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.221124 7274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.221426 7276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:31.224113 7083 server_base.cc:1034] running on GCE node
I20260924 04:03:31.224339 7083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:31.224694 7083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:31.225860 7083 hybrid_clock.cc:648] HybridClock initialized: now 1790222611225853 us; error 32 us; skew 500 ppm
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:31.227552 7083 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:03:31.228106 7083 webserver.cc:460] Webserver started at http://127.0.98.60:35205/ using document root <none> and password file <none>
I20260924 04:03:31.228295 7083 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:31.228334 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:31.229754 7083 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:31.230819 7283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.231053 7083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:31.231168 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
uuid: "fc33482acaf44742a3028d36fbfd81ee"
format_stamp: "Formatted at 2026-09-24 04:03:31 on dist-test-slave-zcsk"
server_key: "3a9dcbcb144484f37b0c7f3a4f0ebb1a"
server_key_iv: "79993da0dd17bf15c4844954c0d8afe9"
server_key_version: "encryptionkey@0"
I20260924 04:03:31.231313 7083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:31.249704 7083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:31.252058 7083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:31.257184 7083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37355
I20260924 04:03:31.257227 7335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37355 every 8 connection(s)
I20260924 04:03:31.257987 7083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:03:31.259955 7336 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:03:31.261523 7336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Bootstrap starting.
I20260924 04:03:31.273334 7336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:03:31.273691 7336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Bootstrap complete.
I20260924 04:03:31.274348 7336 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } }
I20260924 04:03:31.274506 7336 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc33482acaf44742a3028d36fbfd81ee, State: Initialized, Role: LEARNER
I20260924 04:03:31.274619 7336 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } }
I20260924 04:03:31.275012 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.275146 7336 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:31.275230 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: This master's current role is: LEARNER
I20260924 04:03:31.279114 7350 catalog_manager.cc:1259] Loaded cluster ID: 8689bfcb2c00493cb372365e3f424ece
I20260924 04:03:31.279201 7350 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: loading cluster ID for follower catalog manager: success
I20260924 04:03:31.281543 7350 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: acquiring CA information for follower catalog manager: success
I20260924 04:03:31.283303 7350 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 04:03:31.294097 7260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:31.294377 7260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:31.294420 7260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:31.298655 7260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:31.298736 7260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:31.298803 7260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:31.304399 7260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43069
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963,127.0.98.60:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:31.305434 7260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:31.306412 7260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:31.316615 7260 server_base.cc:1034] running on GCE node
W20260924 04:03:31.316530 7355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.316526 7356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.316529 7358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:31.317260 7260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:31.317865 7260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:31.319020 7260 hybrid_clock.cc:648] HybridClock initialized: now 1790222611318997 us; error 34 us; skew 500 ppm
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:31.321619 7260 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:03:31.322448 7260 webserver.cc:460] Webserver started at http://127.0.98.1:43069/ using document root <none> and password file <none>
I20260924 04:03:31.322978 7260 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:31.323055 7260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:31.326792 7260 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:03:31.328786 7365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.330019 7260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:31.330120 7260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "f7c19b464c064f27bab5b704f1ff6127"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "2369c65502fa1e0474bf7395496527ab"
server_key_iv: "b2e704e66f50c88e19848c289325a59c"
server_key_version: "encryptionkey@0"
I20260924 04:03:31.330634 7260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:31.332397 6840 heartbeater.cc:503] Master 127.0.98.62:46295 was elected leader, sending a full tablet report...
I20260924 04:03:31.353834 7260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:31.356987 7260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:31.357165 7260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:31.357761 7260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:31.358633 7260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:31.358680 7260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.358734 7260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:31.358763 7260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.367501 7260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46191
I20260924 04:03:31.367529 7478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46191 every 8 connection(s)
I20260924 04:03:31.368515 7260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:31.371553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 7260
I20260924 04:03:31.371731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 6711
I20260924 04:03:31.378928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45117
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46295
--webserver_port=40259
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963,127.0.98.60:37355 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:31.386243 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.372385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55521 (local address 127.0.98.61:43963)
0924 04:03:31.372511 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.372516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.376820 (+ 4304us) server_negotiation.cc:407] Connection header received
0924 04:03:31.377039 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.377042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.377099 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.377150 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.377676 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.378411 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.379412 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.379733 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.379798 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.382778 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.382782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.382784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.382813 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.384355 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.384848 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.384851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.384853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.384953 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.385297 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.385301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.385302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.385514 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.385584 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.385949 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.386071 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:03:31.386809 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.370355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40105 (local address 127.0.98.62:46295)
0924 04:03:31.370488 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.370493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.376547 (+ 6054us) server_negotiation.cc:407] Connection header received
0924 04:03:31.376653 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.376656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.376711 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.376782 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.377486 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.378275 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.379671 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.380002 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.380068 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.382426 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.382430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.382432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.382459 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.384950 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.385389 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.385392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.385393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.385473 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.385734 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.385737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.385739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.385939 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.386022 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.386564 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.386673 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:03:31.387959 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.370766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:40105)
0924 04:03:31.376452 (+ 5686us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.376465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.376564 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.376965 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.376967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.377196 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.377323 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.377330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.378393 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.378396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.379560 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.379569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.380289 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.380292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.380364 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.380699 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.380712 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.382307 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.385061 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.385064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.385066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.385290 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.385563 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.385566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.385568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.385643 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.386063 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.386066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.386141 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.386225 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.387063 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5591,"mutex_wait_us":24,"spinlock_wait_cycles":37760,"thread_start_us":142,"threads_started":1}
I20260924 04:03:31.387996 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.370326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49113 (local address 127.0.98.60:37355)
0924 04:03:31.370557 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.370566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.371857 (+ 1291us) server_negotiation.cc:407] Connection header received
0924 04:03:31.372814 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.372818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.372925 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.373009 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.374573 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.375332 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.377242 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.377568 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.377665 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.381205 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.381209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.381212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.381245 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.382930 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.383534 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.383538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.383540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.383629 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.384788 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.384792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.384793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.385026 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.385107 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.387671 (+ 2564us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.387795 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260924 04:03:31.388703 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.370766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.98.1:49113)
0924 04:03:31.371331 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.371391 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.372578 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.373229 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.373239 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.373729 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.374407 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.374427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.375462 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.375465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.377115 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.377125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.377835 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.377839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.378048 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.378686 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.378707 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.381009 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.383060 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.383067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.383087 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.383416 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.384462 (+ 1046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.384466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.384468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.384672 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.385154 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.385162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.385509 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.385722 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.386084 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":644,"threads_started":1}
I20260924 04:03:31.388957 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.371890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.98.1:55521)
0924 04:03:31.376745 (+ 4855us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.376758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.376882 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.377277 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.377280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.377454 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.377559 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.377564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.378537 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.378540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.379282 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.379291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.379974 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.379978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.380057 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.380506 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.380521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.382636 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.384472 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.384475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.384478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.384733 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.385065 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.385067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.385069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.385194 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.385655 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.385658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.385759 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.386452 (+ 693us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.388336 (+ 1884us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4790,"mutex_wait_us":1740,"spinlock_wait_cycles":2304,"thread_start_us":76,"threads_started":1}
I20260924 04:03:31.389243 7480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46295
I20260924 04:03:31.389487 7480 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.389858 7481 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43963
I20260924 04:03:31.389956 7481 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.390261 7480 heartbeater.cc:511] Master 127.0.98.62:46295 requested a full tablet report, sending...
I20260924 04:03:31.390287 7479 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37355
I20260924 04:03:31.390388 7479 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.390527 7479 heartbeater.cc:511] Master 127.0.98.60:37355 requested a full tablet report, sending...
I20260924 04:03:31.390262 7481 heartbeater.cc:511] Master 127.0.98.61:43963 requested a full tablet report, sending...
I20260924 04:03:31.391624 6515 ts_manager.cc:194] Re-registered known tserver with Master: f7c19b464c064f27bab5b704f1ff6127 (127.0.98.1:46191)
I20260924 04:03:31.391782 7300 ts_manager.cc:194] Registered new tserver with Master: f7c19b464c064f27bab5b704f1ff6127 (127.0.98.1:46191)
I20260924 04:03:31.391956 7115 ts_manager.cc:194] Registered new tserver with Master: f7c19b464c064f27bab5b704f1ff6127 (127.0.98.1:46191)
I20260924 04:03:31.392365 6515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40105
W20260924 04:03:31.492224 7487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:31.492513 7487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:31.492561 7487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:31.496770 7487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:31.496840 7487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:31.496912 7487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:31.502377 7487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40259
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963,127.0.98.60:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:31.503526 7487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:31.504662 7487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:31.514340 7495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.514290 7492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.514302 7493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:31.514323 7487 server_base.cc:1034] running on GCE node
I20260924 04:03:31.514962 7487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:31.515591 7487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:31.516739 7487 hybrid_clock.cc:648] HybridClock initialized: now 1790222611516712 us; error 41 us; skew 500 ppm
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:31.519583 7487 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:03:31.520598 7487 webserver.cc:460] Webserver started at http://127.0.98.2:40259/ using document root <none> and password file <none>
I20260924 04:03:31.521173 7487 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:31.521247 7487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:31.525310 7487 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:31.527488 7502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.528891 7487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:31.528993 7487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "7ddabd7fc4244e1aaee11067659f6d73"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "c51d8b52304a9d59d9a6c1fbc894326f"
server_key_iv: "1590c74a91c09e36c22569998d86214e"
server_key_version: "encryptionkey@0"
I20260924 04:03:31.529579 7487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:31.544267 6974 heartbeater.cc:503] Master 127.0.98.62:46295 was elected leader, sending a full tablet report...
I20260924 04:03:31.550621 7487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:31.553876 7487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:31.554028 7487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:31.554623 7487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:31.555485 7487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:31.555533 7487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.555572 7487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:31.555596 7487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.564520 7487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45117
I20260924 04:03:31.564546 7615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45117 every 8 connection(s)
I20260924 04:03:31.565661 7487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:31.575707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 7487
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:03:31.575892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 6845
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:03:31.581173 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.567452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59039 (local address 127.0.98.61:43963)
0924 04:03:31.567591 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.567595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.569262 (+ 1667us) server_negotiation.cc:407] Connection header received
0924 04:03:31.569706 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.569709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.569765 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.569842 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.571361 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.571874 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.573379 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.573682 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.573757 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.576997 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.577002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.577004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.577033 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.578787 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.579390 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.579393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.579395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.579497 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.579874 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.579878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.579879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.580138 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.580204 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.580884 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.581007 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:03:31.581502 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.569030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44437 (local address 127.0.98.60:37355)
0924 04:03:31.569156 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.569160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.569917 (+ 757us) server_negotiation.cc:407] Connection header received
0924 04:03:31.570076 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.570079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.570124 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.570202 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.571382 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.572215 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.573189 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.573465 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.573557 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.578126 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.578130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.578133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.578162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.579645 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.580257 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.580260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.580262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.580353 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.580651 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.580654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.580656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.580854 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.580919 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.581242 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.581360 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:03:31.581843 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.567509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58085 (local address 127.0.98.62:46295)
0924 04:03:31.567635 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.567639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.568559 (+ 920us) server_negotiation.cc:407] Connection header received
0924 04:03:31.569699 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.569703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.569765 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.569846 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.571375 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.572230 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.573849 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.574071 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.574165 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.577772 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.577776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.577779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.577809 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.579653 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.580257 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.580260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.580262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.580355 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.580651 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.580654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.580656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.580854 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.580934 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.581589 (+ 655us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.581704 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:03:31.582948 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.568922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.98.2:44437)
0924 04:03:31.569847 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.569859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.569984 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.570219 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.570220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.570598 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.571222 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.571235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.572358 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.572361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.573082 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.573090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.573587 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.573590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.573725 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.574522 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.574538 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.577940 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.579766 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.579769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.579771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.580148 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.580456 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.580458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.580460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.580550 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.580991 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.580994 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.581068 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.581759 (+ 691us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.581892 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":122,"spinlock_wait_cycles":26240,"thread_start_us":76,"threads_started":1}
I20260924 04:03:31.582959 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.567875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:58085)
0924 04:03:31.568391 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.568441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.569487 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.570033 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.570042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.570597 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.571221 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.571235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.572327 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.572331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.573259 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.573269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.574183 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.574186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.574267 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.574695 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.574709 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.577622 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.579766 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.579769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.579771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.580148 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.580450 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.580453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.580455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.580562 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.580974 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.580977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.581071 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.581179 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.582235 (+ 1056us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":164,"spinlock_wait_cycles":44672,"thread_start_us":520,"threads_started":1}
I20260924 04:03:31.583214 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.567875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.98.2:59039)
0924 04:03:31.569173 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.569184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.569508 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.570414 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.570416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.570611 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.571219 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.571233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.572042 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.572045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.573260 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.573269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.573802 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.573805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.573877 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.574486 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.574550 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.576768 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.578913 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.578920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.578938 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.579266 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.579601 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.579603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.579605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.579778 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.580279 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.580286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.580619 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.581431 (+ 812us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.581755 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"mutex_wait_us":108,"spinlock_wait_cycles":11136,"thread_start_us":118,"threads_started":1}
I20260924 04:03:31.584220 7617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46295
I20260924 04:03:31.584270 7616 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37355
I20260924 04:03:31.584285 7618 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43963
I20260924 04:03:31.584477 7618 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.584478 7616 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.584478 7617 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.585001 7618 heartbeater.cc:511] Master 127.0.98.61:43963 requested a full tablet report, sending...
I20260924 04:03:31.585001 7616 heartbeater.cc:511] Master 127.0.98.60:37355 requested a full tablet report, sending...
I20260924 04:03:31.585001 7617 heartbeater.cc:511] Master 127.0.98.62:46295 requested a full tablet report, sending...
I20260924 04:03:31.585518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41799
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46295
--webserver_port=44107
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963,127.0.98.60:37355 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:03:31.586212 7115 ts_manager.cc:194] Registered new tserver with Master: 7ddabd7fc4244e1aaee11067659f6d73 (127.0.98.2:45117)
I20260924 04:03:31.586212 7300 ts_manager.cc:194] Registered new tserver with Master: 7ddabd7fc4244e1aaee11067659f6d73 (127.0.98.2:45117)
I20260924 04:03:31.586989 6515 ts_manager.cc:194] Re-registered known tserver with Master: 7ddabd7fc4244e1aaee11067659f6d73 (127.0.98.2:45117)
I20260924 04:03:31.587558 6515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58085
I20260924 04:03:31.628925 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.625815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.0.1:34490)
0924 04:03:31.625934 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.625949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.626053 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.626288 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.626291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.626310 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:31.626397 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.626402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.627114 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.627117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.627847 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.627856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.628535 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.628538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.628625 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.628647 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.628709 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.628787 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 04:03:31.628925 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.625906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.0.98.60:37355)
0924 04:03:31.626054 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.626059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.626075 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:03:31.626132 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.626135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.626186 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.626261 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:31.626495 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.627010 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.627991 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.628409 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.628509 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.628685 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.628720 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.628785 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:03:31.629580 7310 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 04:03:31.629936 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2e7e0e64f44abf9ea715aa39cbb843. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.630079 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: This master's current role is: LEARNER
W20260924 04:03:31.696695 7623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:31.696979 7623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:31.697042 7623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:31.701049 7623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:31.701107 7623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:31.701182 7623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:31.706401 7623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44107
--tserver_master_addrs=127.0.98.62:46295,127.0.98.61:43963,127.0.98.60:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:31.707469 7623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:31.708580 7623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:31.718396 7628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.718462 7631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:31.718407 7629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:31.718446 7623 server_base.cc:1034] running on GCE node
I20260924 04:03:31.719133 7623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:31.719803 7623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:31.720969 7623 hybrid_clock.cc:648] HybridClock initialized: now 1790222611720947 us; error 33 us; skew 500 ppm
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:31.723887 7623 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:03:31.724838 7623 webserver.cc:460] Webserver started at http://127.0.98.3:44107/ using document root <none> and password file <none>
I20260924 04:03:31.725420 7623 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:31.725507 7623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:31.729686 7623 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:03:31.732374 7638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.733850 7623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:31.733985 7623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "4adba512c0ac497b88071971d6c71848"
format_stamp: "Formatted at 2026-09-24 04:03:30 on dist-test-slave-zcsk"
server_key: "09463fc22f3280352f30a6578896459e"
server_key_iv: "3483c753742677ae2dab904deeaad56b"
server_key_version: "encryptionkey@0"
I20260924 04:03:31.734598 7623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:31.763600 7623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:31.767010 7623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:31.767194 7623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:31.767788 7623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:31.768678 7623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:31.768726 7623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.768795 7623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:31.768821 7623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:31.777364 7623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41799
I20260924 04:03:31.777401 7751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41799 every 8 connection(s)
I20260924 04:03:31.778398 7623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:31.782015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 7623
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:03:31 dist-test-slave-zcsk krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:03:31.797180 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.780276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58171 (local address 127.0.98.62:46295)
0924 04:03:31.780398 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.780402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.781308 (+ 906us) server_negotiation.cc:407] Connection header received
0924 04:03:31.783561 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.783565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.783629 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.783686 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.785640 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.786472 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.788914 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.789272 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.789366 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.792683 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.792687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.792689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.792720 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.794638 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.795842 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.795846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.795847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.795975 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.796268 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.796271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.796273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.796472 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.796546 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.796897 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.797006 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":49}
I20260924 04:03:31.798352 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.780655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.98.3:45813)
0924 04:03:31.781441 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.781454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.783305 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.784045 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.784053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.784644 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.785485 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.785506 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.786592 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.786595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.788173 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.788183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.788945 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.788949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.789144 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.790283 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.790308 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.792255 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.794917 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.794920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.794922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.795211 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.795517 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.795520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.795522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.795707 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.796228 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.796234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.796586 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.796818 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.797195 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":227,"spinlock_wait_cycles":25856,"thread_start_us":119,"threads_started":1}
I20260924 04:03:31.798627 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.780655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:58171)
0924 04:03:31.781140 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.781189 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.783302 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.784044 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.784053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.785301 (+ 1248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.785501 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.785509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.786594 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.786596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.788812 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.788821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.789771 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.789774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.789876 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.790342 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.790354 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.792514 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.794769 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.794776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.794794 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.795730 (+ 936us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.796085 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.796088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.796090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.796169 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.796600 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.796603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.796687 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.797404 (+ 717us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.797536 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":1095,"thread_start_us":531,"threads_started":1}
I20260924 04:03:31.799058 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.780213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45813 (local address 127.0.98.61:43963)
0924 04:03:31.780362 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.780367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.781528 (+ 1161us) server_negotiation.cc:407] Connection header received
0924 04:03:31.783738 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.783744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.783797 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.783864 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.785646 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.786471 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.788320 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.788670 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.788770 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.792459 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.792463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.792465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.792490 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.794784 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.795327 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.795330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.795332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.795415 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.795831 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.795834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.795836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.796082 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.796173 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.798787 (+ 2614us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.798901 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":70}
I20260924 04:03:31.799248 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.784802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.0.98.60:37355)
0924 04:03:31.785186 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.785190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.786892 (+ 1702us) server_negotiation.cc:407] Connection header received
0924 04:03:31.786933 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.786936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.786981 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.787054 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.787892 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.788665 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.790119 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.790477 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.790561 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.793852 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.793856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.793859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.793898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.796534 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.797406 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.797409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.797411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.797492 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.797768 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.797771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.797773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.797964 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.798076 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.798964 (+ 888us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.799088 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":58,"threads_started":1}
I20260924 04:03:31.799726 7753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46295
I20260924 04:03:31.800031 7753 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.800089 7754 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43963
I20260924 04:03:31.800169 7754 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.800652 7754 heartbeater.cc:511] Master 127.0.98.61:43963 requested a full tablet report, sending...
I20260924 04:03:31.800652 7753 heartbeater.cc:511] Master 127.0.98.62:46295 requested a full tablet report, sending...
I20260924 04:03:31.801443 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.784325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.0.98.62:46295)
0924 04:03:31.785844 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.785854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.785866 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:03:31.785900 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.785902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.785945 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.786867 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.786963 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.787719 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.788895 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.790509 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.793611 (+ 3102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.793728 (+ 117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.793731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.793734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.793753 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.796969 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.799402 (+ 2433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.799406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.799407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.799513 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.800187 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.800190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.800192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.800396 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.801092 (+ 696us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.801163 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.801284 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":1443,"thread_start_us":91,"threads_started":1}
I20260924 04:03:31.801800 6515 ts_manager.cc:194] Re-registered known tserver with Master: 4adba512c0ac497b88071971d6c71848 (127.0.98.3:41799)
I20260924 04:03:31.801954 7115 ts_manager.cc:194] Registered new tserver with Master: 4adba512c0ac497b88071971d6c71848 (127.0.98.3:41799)
I20260924 04:03:31.802435 6515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58171
I20260924 04:03:31.803369 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.784553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.0.98.61:43963)
0924 04:03:31.785057 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.785061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.785536 (+ 475us) server_negotiation.cc:407] Connection header received
0924 04:03:31.785577 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.785580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.785627 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.785697 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.788949 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.789708 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.790803 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.791148 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.791213 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.793457 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.793461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.793463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.793489 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.798118 (+ 4629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.800425 (+ 2307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.800428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.800430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.800514 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.802096 (+ 1582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.802099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.802101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.802320 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.802437 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.803046 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.803208 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":436,"thread_start_us":68,"threads_started":1}
I20260924 04:03:31.804225 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.781655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.98.3:38937)
0924 04:03:31.781878 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.781891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.783305 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.784043 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.784053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.784630 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:31.785485 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.785506 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.786590 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.786605 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.790086 (+ 3481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.790095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.791511 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.791514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.791604 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.792768 (+ 1164us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:31.792781 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:31.795446 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:31.798242 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.798245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.798247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.799032 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.800626 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:31.800629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:31.800631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:31.800715 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:31.801218 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:31.801221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:31.801330 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.803834 (+ 2504us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.804037 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":2488,"spinlock_wait_cycles":9856,"thread_start_us":77,"threads_started":1}
I20260924 04:03:31.804517 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.781760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38937 (local address 127.0.98.60:37355)
0924 04:03:31.781877 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.781882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.781956 (+ 74us) server_negotiation.cc:407] Connection header received
0924 04:03:31.783554 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.783557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.783616 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.783670 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.785646 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.786414 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.790985 (+ 4571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.791283 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.791347 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.795597 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.795600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.795602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.795624 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:31.797089 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.799885 (+ 2796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.799889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.799890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.800263 (+ 373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:31.800891 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:31.800894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:31.800896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:31.801089 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:31.801164 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.802968 (+ 1804us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.803326 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:03:31.805013 7752 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37355
I20260924 04:03:31.805102 7752 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:31.805287 7752 heartbeater.cc:511] Master 127.0.98.60:37355 requested a full tablet report, sending...
I20260924 04:03:31.805917 7300 ts_manager.cc:194] Registered new tserver with Master: 4adba512c0ac497b88071971d6c71848 (127.0.98.3:41799)
I20260924 04:03:31.804492 6515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:31.807991 6515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:31.810386 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: true } } }
I20260924 04:03:31.810503 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:31.813875 7264 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: attempting to promote NON_VOTER fc33482acaf44742a3028d36fbfd81ee to VOTER
I20260924 04:03:31.814145 7264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } }
I20260924 04:03:31.815208 7310 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 LEARNER]: Refusing update from remote peer 4e2e7e0e64f44abf9ea715aa39cbb843: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:03:31.815326 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Refusing update from remote peer 4e2e7e0e64f44abf9ea715aa39cbb843: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:03:31.815603 7264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:03:31.815753 7771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:03:31.818645 7264 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc33482acaf44742a3028d36fbfd81ee (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:31.819808 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:31.819934 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:31.820062 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:31.820120 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:31.820245 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:31.820128 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:31.821121 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc33482acaf44742a3028d36fbfd81ee (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:31.821156 7310 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fc33482acaf44742a3028d36fbfd81ee (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:31.823099 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:31.823220 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:03:31.823762 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:31.823887 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:03:31.831120 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.826311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.0.98.3:41799)
0924 04:03:31.826609 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.826620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.826642 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:03:31.826707 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.826711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.826903 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.827056 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:31.827248 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.828120 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.829941 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.830544 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.830642 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.830854 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.830904 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.830996 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260924 04:03:31.831203 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.825151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45117 (local address 127.0.0.1:36368)
0924 04:03:31.825596 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.825612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.825694 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.827312 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.827314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.827329 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:31.827392 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.827397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.828649 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.828652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.829825 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.829831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.830784 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.830788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.830882 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.830904 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.830952 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.831020 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":65}
I20260924 04:03:31.831250 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.825354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46191 (local address 127.0.0.1:39130)
0924 04:03:31.826148 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.826162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.826247 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.827001 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.827005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.827021 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:31.827124 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.827130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.828681 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.828684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.829655 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.829664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.830917 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.830920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.830996 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.831014 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.831068 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.831125 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":25,"thread_start_us":94,"threads_started":1}
I20260924 04:03:31.831068 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.825930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41799 (local address 127.0.0.1:56438)
0924 04:03:31.826465 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.826477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.826556 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.827050 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.827053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.827070 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:31.827163 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.827169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.828218 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.828221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.829804 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.829813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.830653 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.830656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.830737 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.830761 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.830818 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.830897 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260924 04:03:31.831652 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.825293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.0.98.2:45117)
0924 04:03:31.826931 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.826941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.826963 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:03:31.827031 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.827036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.827204 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.827340 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:31.827472 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.828555 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.829926 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.830647 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.831193 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.831322 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.831374 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.831465 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":132,"threads_started":1}
I20260924 04:03:31.831974 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.825690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.98.1:46191)
0924 04:03:31.825994 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.826007 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.826240 (+ 233us) server_negotiation.cc:407] Connection header received
0924 04:03:31.826408 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.826416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.826671 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.826835 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:31.827242 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.828555 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.829774 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.830613 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.830720 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.830864 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.831404 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.831489 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
I20260924 04:03:31.834915 7413 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8100f3ae924456b5a7707e09f85644 (DEFAULT_TABLE table=test-table [id=65cfa184c1aa482283ce42f6201dfead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:31.834923 7686 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8100f3ae924456b5a7707e09f85644 (DEFAULT_TABLE table=test-table [id=65cfa184c1aa482283ce42f6201dfead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:31.834920 7550 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8100f3ae924456b5a7707e09f85644 (DEFAULT_TABLE table=test-table [id=65cfa184c1aa482283ce42f6201dfead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:31.835966 7413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8100f3ae924456b5a7707e09f85644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:31.835966 7686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8100f3ae924456b5a7707e09f85644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:31.836723 7550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8100f3ae924456b5a7707e09f85644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:31.842747 7786 tablet_bootstrap.cc:492] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Bootstrap starting.
I20260924 04:03:31.843788 7788 tablet_bootstrap.cc:492] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: Bootstrap starting.
I20260924 04:03:31.845661 7786 tablet_bootstrap.cc:654] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:31.845861 7788 tablet_bootstrap.cc:654] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:31.846560 7786 log.cc:824] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:31.846560 7788 log.cc:824] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:31.847256 7787 tablet_bootstrap.cc:492] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: Bootstrap starting.
I20260924 04:03:31.848672 7788 tablet_bootstrap.cc:492] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: No bootstrap required, opened a new log
I20260924 04:03:31.848701 7786 tablet_bootstrap.cc:492] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: No bootstrap required, opened a new log
I20260924 04:03:31.848865 7786 ts_tablet_manager.cc:1397] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:03:31.848863 7788 ts_tablet_manager.cc:1397] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:03:31.849354 7787 tablet_bootstrap.cc:654] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:31.850149 7787 log.cc:824] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:31.852234 7787 tablet_bootstrap.cc:492] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: No bootstrap required, opened a new log
I20260924 04:03:31.852442 7787 ts_tablet_manager.cc:1397] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:03:31.852778 7788 raft_consensus.cc:348] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.853005 7788 raft_consensus.cc:374] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:31.853062 7788 raft_consensus.cc:729] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ddabd7fc4244e1aaee11067659f6d73, State: Initialized, Role: FOLLOWER
I20260924 04:03:31.853148 7786 raft_consensus.cc:348] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.853395 7786 raft_consensus.cc:374] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:31.853441 7786 raft_consensus.cc:729] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c19b464c064f27bab5b704f1ff6127, State: Initialized, Role: FOLLOWER
I20260924 04:03:31.853499 7788 consensus_queue.cc:260] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.853858 7786 consensus_queue.cc:260] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.854494 7788 ts_tablet_manager.cc:1428] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 04:03:31.854830 7617 heartbeater.cc:503] Master 127.0.98.62:46295 was elected leader, sending a full tablet report...
I20260924 04:03:31.854832 7786 ts_tablet_manager.cc:1428] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:03:31.856186 7480 heartbeater.cc:503] Master 127.0.98.62:46295 was elected leader, sending a full tablet report...
I20260924 04:03:31.859970 7787 raft_consensus.cc:348] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.860245 7787 raft_consensus.cc:374] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:31.860311 7787 raft_consensus.cc:729] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4adba512c0ac497b88071971d6c71848, State: Initialized, Role: FOLLOWER
I20260924 04:03:31.860773 7787 consensus_queue.cc:260] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.861725 7787 ts_tablet_manager.cc:1428] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 04:03:31.862025 7753 heartbeater.cc:503] Master 127.0.98.62:46295 was elected leader, sending a full tablet report...
W20260924 04:03:31.870918 7482 tablet.cc:2369] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:31.920149 7793 raft_consensus.cc:482] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:31.920341 7793 raft_consensus.cc:504] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.921425 7793 leader_election.cc:288] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ddabd7fc4244e1aaee11067659f6d73 (127.0.98.2:45117), 4adba512c0ac497b88071971d6c71848 (127.0.98.3:41799)
I20260924 04:03:31.925204 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.921715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53693 (local address 127.0.98.2:45117)
0924 04:03:31.921834 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.921839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.922076 (+ 237us) server_negotiation.cc:407] Connection header received
0924 04:03:31.922191 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.922194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.922283 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.922348 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:31.922612 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.923123 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.924265 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.924685 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.924779 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.924837 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.925014 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.925084 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:03:31.925211 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.921777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51203 (local address 127.0.98.3:41799)
0924 04:03:31.921895 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.921900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.922138 (+ 238us) server_negotiation.cc:407] Connection header received
0924 04:03:31.922308 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.922310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.922362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.922430 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:31.922658 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.923200 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.924264 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.924670 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.924753 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.924817 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.925040 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.925111 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:03:31.925215 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.921628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45117 (local address 127.0.98.1:53693)
0924 04:03:31.921983 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.921996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.922100 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.922388 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.922391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.922407 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:31.922516 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.922522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.923224 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.923227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.924153 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.924162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.924798 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.924801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.924886 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.924925 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.924985 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.925060 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":36,"thread_start_us":83,"threads_started":1}
I20260924 04:03:31.925212 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.921670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41799 (local address 127.0.98.1:51203)
0924 04:03:31.922062 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:31.922076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:31.922202 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:31.922450 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:31.922453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:31.922469 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:31.922555 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.922560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.923282 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.923285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.924155 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:31.924164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.924777 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.924781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:31.924874 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.924934 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 04:03:31.924999 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:03:31.925070 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":42,"thread_start_us":75,"threads_started":1}
I20260924 04:03:31.925994 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8100f3ae924456b5a7707e09f85644" candidate_uuid: "f7c19b464c064f27bab5b704f1ff6127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ddabd7fc4244e1aaee11067659f6d73" is_pre_election: true
I20260924 04:03:31.926074 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8100f3ae924456b5a7707e09f85644" candidate_uuid: "f7c19b464c064f27bab5b704f1ff6127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4adba512c0ac497b88071971d6c71848" is_pre_election: true
I20260924 04:03:31.926281 7570 raft_consensus.cc:2454] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c19b464c064f27bab5b704f1ff6127 in term 0.
I20260924 04:03:31.926324 7706 raft_consensus.cc:2454] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c19b464c064f27bab5b704f1ff6127 in term 0.
I20260924 04:03:31.926738 7368 leader_election.cc:302] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4adba512c0ac497b88071971d6c71848, f7c19b464c064f27bab5b704f1ff6127; no voters:
I20260924 04:03:31.926997 7793 raft_consensus.cc:2781] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:31.927073 7793 raft_consensus.cc:482] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:31.927103 7793 raft_consensus.cc:3037] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:31.928215 7793 raft_consensus.cc:504] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.928570 7793 leader_election.cc:288] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [CANDIDATE]: Term 1 election: Requested vote from peers 7ddabd7fc4244e1aaee11067659f6d73 (127.0.98.2:45117), 4adba512c0ac497b88071971d6c71848 (127.0.98.3:41799)
I20260924 04:03:31.928894 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8100f3ae924456b5a7707e09f85644" candidate_uuid: "f7c19b464c064f27bab5b704f1ff6127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4adba512c0ac497b88071971d6c71848"
I20260924 04:03:31.929040 7706 raft_consensus.cc:3037] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:31.930023 7706 raft_consensus.cc:2454] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c19b464c064f27bab5b704f1ff6127 in term 1.
I20260924 04:03:31.930264 7570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8100f3ae924456b5a7707e09f85644" candidate_uuid: "f7c19b464c064f27bab5b704f1ff6127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ddabd7fc4244e1aaee11067659f6d73"
I20260924 04:03:31.930303 7368 leader_election.cc:302] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4adba512c0ac497b88071971d6c71848, f7c19b464c064f27bab5b704f1ff6127; no voters:
I20260924 04:03:31.930404 7570 raft_consensus.cc:3037] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:31.930454 7793 raft_consensus.cc:2781] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:31.930665 7793 raft_consensus.cc:686] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 1 LEADER]: Becoming Leader. State: Replica: f7c19b464c064f27bab5b704f1ff6127, State: Running, Role: LEADER
I20260924 04:03:31.931210 7793 consensus_queue.cc:237] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } }
I20260924 04:03:31.931687 7570 raft_consensus.cc:2454] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c19b464c064f27bab5b704f1ff6127 in term 1.
I20260924 04:03:31.934767 6514 catalog_manager.cc:5495] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c19b464c064f27bab5b704f1ff6127 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f7c19b464c064f27bab5b704f1ff6127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } health_report { overall_health: UNKNOWN } } }
I20260924 04:03:31.953738 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.950333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.0.98.1:46191)
0924 04:03:31.950489 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.950493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.950506 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:31.950580 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.950582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.950637 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.950723 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:31.950953 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.951462 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.952318 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.952539 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.952622 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.952913 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:31.953358 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:31.953491 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:31.953556 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 04:03:31.953612 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:03:31.959427 7570 raft_consensus.cc:1261] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 FOLLOWER]: Refusing update from remote peer f7c19b464c064f27bab5b704f1ff6127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:31.959427 7706 raft_consensus.cc:1261] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 1 FOLLOWER]: Refusing update from remote peer f7c19b464c064f27bab5b704f1ff6127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:31.960206 7793 consensus_queue.cc:1035] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:31.960402 7797 consensus_queue.cc:1035] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:31.971815 7802 mvcc.cc:205] Tried to move back new op lower bound from 7332751818578558976 to 7332751818473816064. Current Snapshot: MvccSnapshot[applied={T|T < 7332751818578558976 or (T in {7332751818578558976})}]
I20260924 04:03:31.981364 6515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:31.981586 6515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:31.987494 6515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e2e7e0e64f44abf9ea715aa39cbb843: Sending DeleteTablet for 3 replicas of tablet 7f8100f3ae924456b5a7707e09f85644
I20260924 04:03:31.988283 7550 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8100f3ae924456b5a7707e09f85644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36368
I20260924 04:03:31.988349 7413 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8100f3ae924456b5a7707e09f85644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39130
I20260924 04:03:31.988833 7811 tablet_replica.cc:330] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: stopping tablet replica
I20260924 04:03:31.989059 7686 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8100f3ae924456b5a7707e09f85644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56438
I20260924 04:03:31.989192 7811 raft_consensus.cc:2229] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:31.989540 7811 raft_consensus.cc:2258] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:31.989830 7812 tablet_replica.cc:330] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: stopping tablet replica
I20260924 04:03:31.990073 7810 tablet_replica.cc:330] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: stopping tablet replica
I20260924 04:03:31.990130 7812 raft_consensus.cc:2229] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:31.990330 7810 raft_consensus.cc:2229] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:31.990424 7812 raft_consensus.cc:2258] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:31.990728 7810 raft_consensus.cc:2258] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:31.991334 7811 ts_tablet_manager.cc:1905] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:31.992043 7810 ts_tablet_manager.cc:1905] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:31.992172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 6483
I20260924 04:03:31.995294 7810 ts_tablet_manager.cc:1918] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:31.995396 7810 log.cc:1196] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/7f8100f3ae924456b5a7707e09f85644
I20260924 04:03:31.995688 7810 ts_tablet_manager.cc:1939] T 7f8100f3ae924456b5a7707e09f85644 P f7c19b464c064f27bab5b704f1ff6127: Deleting consensus metadata
W20260924 04:03:31.998755 7368 connection.cc:535] server connection from 127.0.0.1:39130 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:03:32.000590 7812 ts_tablet_manager.cc:1905] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:32.002784 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.991141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.0.98.61:43963)
0924 04:03:31.991262 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.991266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:31.993039 (+ 1773us) server_negotiation.cc:407] Connection header received
0924 04:03:31.993220 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:31.993223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:31.993273 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:31.993321 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:31.994310 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.995181 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.996616 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:31.996990 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:31.997062 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:31.999088 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:31.999093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:31.999096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:31.999125 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:32.000833 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:32.001417 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:32.001421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:32.001423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:32.001511 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:32.001812 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:32.001815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:32.001817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:32.002031 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:32.002130 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:32.002495 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 04:03:32.002625 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:03:32.003448 7812 ts_tablet_manager.cc:1918] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:32.003598 7812 log.cc:1196] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/7f8100f3ae924456b5a7707e09f85644
I20260924 04:03:32.003983 7812 ts_tablet_manager.cc:1939] T 7f8100f3ae924456b5a7707e09f85644 P 4adba512c0ac497b88071971d6c71848: Deleting consensus metadata
I20260924 04:03:32.005167 7811 ts_tablet_manager.cc:1918] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 04:03:32.005203 7639 connection.cc:441] server connection from 127.0.0.1:56438 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56438 (request call id 1) could send its response
I20260924 04:03:32.005285 7811 log.cc:1196] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/7f8100f3ae924456b5a7707e09f85644
I20260924 04:03:32.005610 7811 ts_tablet_manager.cc:1939] T 7f8100f3ae924456b5a7707e09f85644 P 7ddabd7fc4244e1aaee11067659f6d73: Deleting consensus metadata
W20260924 04:03:32.006723 7504 connection.cc:441] server connection from 127.0.0.1:36368 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36368 (request call id 1) could send its response
I20260924 04:03:32.012996 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:31.990653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.0.98.60:37355)
0924 04:03:31.990778 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:31.990783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:32.003171 (+ 12388us) server_negotiation.cc:407] Connection header received
0924 04:03:32.003394 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:32.003398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:32.003451 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:32.003500 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:32.004549 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:32.005323 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:32.006440 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:32.006671 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:32.006775 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:32.008436 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:32.008440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:32.008443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:32.008471 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:32.010184 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:32.010647 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:32.010650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:32.010652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:32.010736 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:32.011032 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:32.011035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:32.011037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:32.011518 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:32.011646 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:32.012686 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0924 04:03:32.012812 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:32.015594 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:32.005572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.0.98.61:43963)
0924 04:03:32.005663 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:32.005667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:32.006855 (+ 1188us) server_negotiation.cc:407] Connection header received
0924 04:03:32.007013 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:32.007016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:32.007056 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:32.007113 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:32.008271 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:32.009004 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:32.010357 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:32.010669 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:32.010739 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:32.012269 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:32.012273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:32.012276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:32.012302 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:32.013910 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:32.014387 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:32.014391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:32.014393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:32.014483 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:32.014742 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:32.014745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:32.014747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:32.014972 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:32.015089 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:32.015310 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 04:03:32.015432 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:03:32.020778 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:32.005361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.0.98.60:37355)
0924 04:03:32.005496 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:32.005499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:32.008603 (+ 3104us) server_negotiation.cc:407] Connection header received
0924 04:03:32.008745 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:32.008748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:32.008791 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:32.008835 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:32.009585 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:32.010339 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:32.011390 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:32.011729 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:32.011809 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:32.017239 (+ 5430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:32.017243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:32.017246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:32.017272 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:32.018936 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:32.019453 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:32.019457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:32.019459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:32.019542 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:32.019794 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:32.019797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:32.019799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:32.020174 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:32.020290 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:32.020504 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:03:32.020623 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:03:32.972932 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:32.972415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:57567)
0924 04:03:32.972658 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:32.972741 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
W20260924 04:03:32.973286 7753 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:32.975595 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:32.975199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:46171)
0924 04:03:32.975427 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:32.975506 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
W20260924 04:03:32.976329 7480 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:32.976795 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:32.976217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:50527)
0924 04:03:32.976601 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:32.976711 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":63,"threads_started":1}
W20260924 04:03:32.977049 7617 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:33.487306 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e2e7e0e64f44abf9ea715aa39cbb843)
I20260924 04:03:33.487501 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } }
I20260924 04:03:33.488059 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e2e7e0e64f44abf9ea715aa39cbb843)
I20260924 04:03:33.488245 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } }
I20260924 04:03:33.488543 7824 leader_election.cc:288] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295), 0b7890dab2c943f9bb8610647603ee18 (127.0.98.61:43963)
I20260924 04:03:33.489262 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.488743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60636)
0924 04:03:33.489061 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.489150 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":68,"threads_started":1}
I20260924 04:03:33.489251 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295), fc33482acaf44742a3028d36fbfd81ee (127.0.98.60:37355)
W20260924 04:03:33.489442 7284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:33.489871 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.489312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60646)
0924 04:03:33.489710 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.489780 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":50,"threads_started":1}
W20260924 04:03:33.490026 7099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:33.490299 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.490059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60662)
0924 04:03:33.490192 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.490237 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 04:03:33.490432 7284 leader_election.cc:334] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:33.490909 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.490679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60666)
0924 04:03:33.490805 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.490851 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 04:03:33.491014 7099 leader_election.cc:334] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:33.492933 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.488764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43963 (local address 127.0.0.1:34810)
0924 04:03:33.489091 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.489105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:33.489237 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:33.489964 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:33.489967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:33.489987 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:33.490099 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:33.490106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:33.491097 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:33.491100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:33.491951 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:33.491961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:33.492558 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:33.492562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:33.492648 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:33.492675 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:03:33.492738 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:03:33.492810 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260924 04:03:33.492929 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.488848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.0.98.61:43963)
0924 04:03:33.489099 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:33.489103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:33.489721 (+ 618us) server_negotiation.cc:407] Connection header received
0924 04:03:33.489781 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:33.489784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:33.489834 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:33.490201 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:33.490298 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:33.491019 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:33.492063 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:33.492461 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:33.492548 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:33.492605 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:33.492759 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:03:33.492823 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260924 04:03:33.493525 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.489523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.0.98.60:37355)
0924 04:03:33.490452 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:33.490456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:33.490469 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:33.490510 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:33.490513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:33.490551 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:33.490667 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:33.490907 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:33.491614 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:33.492654 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:33.493090 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:33.493200 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:33.493251 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:33.493388 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 04:03:33.493444 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":45,"threads_started":1}
I20260924 04:03:33.493537 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.489399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37355 (local address 127.0.0.1:34520)
0924 04:03:33.489835 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.489846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:33.490027 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:33.490700 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:33.490703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:33.490721 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:33.490799 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:33.490804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:33.491723 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:33.491727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:33.492546 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:33.492555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:33.493198 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:33.493202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:33.493285 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:33.493302 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:03:33.493368 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:03:33.493431 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":60,"threads_started":1}
I20260924 04:03:33.493494 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc33482acaf44742a3028d36fbfd81ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b7890dab2c943f9bb8610647603ee18" is_pre_election: true
I20260924 04:03:33.493695 7125 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc33482acaf44742a3028d36fbfd81ee in term 1.
I20260924 04:03:33.493965 7310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b7890dab2c943f9bb8610647603ee18" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc33482acaf44742a3028d36fbfd81ee" is_pre_election: true
I20260924 04:03:33.494004 7285 leader_election.cc:302] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b7890dab2c943f9bb8610647603ee18, fc33482acaf44742a3028d36fbfd81ee; no voters: 4e2e7e0e64f44abf9ea715aa39cbb843
I20260924 04:03:33.494130 7310 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7890dab2c943f9bb8610647603ee18 in term 1.
I20260924 04:03:33.494205 7824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:03:33.494261 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e2e7e0e64f44abf9ea715aa39cbb843)
I20260924 04:03:33.494304 7824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:03:33.494465 7100 leader_election.cc:302] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b7890dab2c943f9bb8610647603ee18, fc33482acaf44742a3028d36fbfd81ee; no voters: 4e2e7e0e64f44abf9ea715aa39cbb843
I20260924 04:03:33.494647 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:03:33.494700 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e2e7e0e64f44abf9ea715aa39cbb843)
I20260924 04:03:33.494742 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:03:33.495345 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } }
I20260924 04:03:33.495671 7824 leader_election.cc:288] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 2 election: Requested vote from peers 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295), 0b7890dab2c943f9bb8610647603ee18 (127.0.98.61:43963)
I20260924 04:03:33.495649 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } }
I20260924 04:03:33.495975 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc33482acaf44742a3028d36fbfd81ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b7890dab2c943f9bb8610647603ee18"
I20260924 04:03:33.496088 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.495793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60668)
0924 04:03:33.495968 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.496025 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 04:03:33.496151 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [CANDIDATE]: Term 2 election: Requested vote from peers 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295), fc33482acaf44742a3028d36fbfd81ee (127.0.98.60:37355)
I20260924 04:03:33.496174 7125 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fc33482acaf44742a3028d36fbfd81ee in current term 2: Already voted for candidate 0b7890dab2c943f9bb8610647603ee18 in this term.
I20260924 04:03:33.496382 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.496135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60684)
0924 04:03:33.496275 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.496322 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 04:03:33.496709 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.496500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60694)
0924 04:03:33.496610 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.496655 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 04:03:33.496692 7310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b7890dab2c943f9bb8610647603ee18" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc33482acaf44742a3028d36fbfd81ee"
I20260924 04:03:33.496814 7310 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b7890dab2c943f9bb8610647603ee18 in current term 2: Already voted for candidate fc33482acaf44742a3028d36fbfd81ee in this term.
W20260924 04:03:33.496811 7284 leader_election.cc:334] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:33.496901 7284 leader_election.cc:302] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc33482acaf44742a3028d36fbfd81ee; no voters: 0b7890dab2c943f9bb8610647603ee18, 4e2e7e0e64f44abf9ea715aa39cbb843
I20260924 04:03:33.496955 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.496749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60698)
0924 04:03:33.496870 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.496908 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 04:03:33.497018 7099 leader_election.cc:334] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:33.497112 7824 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 04:03:33.497159 7100 leader_election.cc:302] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7890dab2c943f9bb8610647603ee18; no voters: 4e2e7e0e64f44abf9ea715aa39cbb843, fc33482acaf44742a3028d36fbfd81ee
I20260924 04:03:33.497283 7825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 04:03:33.974366 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.973904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:37803)
0924 04:03:33.974214 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.974272 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260924 04:03:33.977684 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.976919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:44101)
0924 04:03:33.977437 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.977500 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":96,"threads_started":1}
I20260924 04:03:33.977927 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:33.977484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:38161)
0924 04:03:33.977785 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:33.977842 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":68,"threads_started":1}
I20260924 04:03:34.975724 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:34.975239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:45983)
0924 04:03:34.975524 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:34.975614 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260924 04:03:34.979149 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:34.978594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:52945)
0924 04:03:34.978949 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:34.979016 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":63,"threads_started":1}
I20260924 04:03:34.979159 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:34.978378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:38867)
0924 04:03:34.978972 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:34.979039 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":71,"threads_started":1}
I20260924 04:03:35.438611 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:35.438751 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } }
I20260924 04:03:35.439240 7840 leader_election.cc:288] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295), 0b7890dab2c943f9bb8610647603ee18 (127.0.98.61:43963)
I20260924 04:03:35.439711 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc33482acaf44742a3028d36fbfd81ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b7890dab2c943f9bb8610647603ee18" is_pre_election: true
I20260924 04:03:35.439790 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.439416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60980)
0924 04:03:35.439641 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.439704 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
I20260924 04:03:35.439851 7125 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc33482acaf44742a3028d36fbfd81ee in term 2.
I20260924 04:03:35.440179 7285 leader_election.cc:302] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7890dab2c943f9bb8610647603ee18, fc33482acaf44742a3028d36fbfd81ee; no voters:
I20260924 04:03:35.440361 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 04:03:35.440419 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:35.440446 7840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:03:35.440706 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.440469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:60986)
0924 04:03:35.440610 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.440652 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 04:03:35.440807 7284 leader_election.cc:334] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:35.441579 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } }
I20260924 04:03:35.441874 7840 leader_election.cc:288] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 3 election: Requested vote from peers 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295), 0b7890dab2c943f9bb8610647603ee18 (127.0.98.61:43963)
I20260924 04:03:35.442195 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.441986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:32770)
0924 04:03:35.442077 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.442120 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 04:03:35.442261 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc33482acaf44742a3028d36fbfd81ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b7890dab2c943f9bb8610647603ee18"
I20260924 04:03:35.442391 7125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:03:35.442788 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.442598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:32780)
0924 04:03:35.442689 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.442735 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:03:35.442879 7284 leader_election.cc:334] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
I20260924 04:03:35.443436 7125 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc33482acaf44742a3028d36fbfd81ee in term 3.
I20260924 04:03:35.443738 7285 leader_election.cc:302] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b7890dab2c943f9bb8610647603ee18, fc33482acaf44742a3028d36fbfd81ee; no voters: 4e2e7e0e64f44abf9ea715aa39cbb843
I20260924 04:03:35.443893 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:03:35.444135 7840 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [term 3 LEADER]: Becoming Leader. State: Replica: fc33482acaf44742a3028d36fbfd81ee, State: Running, Role: LEADER
I20260924 04:03:35.444377 7840 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } }
I20260924 04:03:35.445709 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc33482acaf44742a3028d36fbfd81ee. Latest consensus state: current_term: 3 leader_uuid: "fc33482acaf44742a3028d36fbfd81ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:35.445832 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:35.446147 7845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:35.448360 7845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:35.448781 7845 catalog_manager.cc:1259] Loaded cluster ID: 8689bfcb2c00493cb372365e3f424ece
I20260924 04:03:35.448835 7845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:35.449363 7845 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:35.449717 7845 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Loaded TSK: 0
I20260924 04:03:35.450350 7845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:03:35.466176 7299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:35.469174 7299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:35.472148 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.471884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:32800)
0924 04:03:35.472012 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.472075 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:03:35.472209 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [term 3 FOLLOWER]: Refusing update from remote peer fc33482acaf44742a3028d36fbfd81ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 04:03:35.472666 7843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 04:03:35.472847 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.472639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.0.1:32812)
0924 04:03:35.472739 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.472789 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:03:35.472942 7284 consensus_peers.cc:487] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee -> Peer 4e2e7e0e64f44abf9ea715aa39cbb843 (127.0.98.62:46295): Couldn't send request to peer 4e2e7e0e64f44abf9ea715aa39cbb843. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:03:35.474138 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc33482acaf44742a3028d36fbfd81ee. Latest consensus state: current_term: 3 leader_uuid: "fc33482acaf44742a3028d36fbfd81ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:35.474258 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:03:35.475211 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fc33482acaf44742a3028d36fbfd81ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:35.475350 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:35.475776 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fc33482acaf44742a3028d36fbfd81ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e2e7e0e64f44abf9ea715aa39cbb843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46295 } } peers { permanent_uuid: "0b7890dab2c943f9bb8610647603ee18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43963 } attrs { promote: false } } peers { permanent_uuid: "fc33482acaf44742a3028d36fbfd81ee" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37355 } attrs { promote: false } } }
I20260924 04:03:35.475888 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b7890dab2c943f9bb8610647603ee18 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:03:35.485708 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.481393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.0.98.3:41799)
0924 04:03:35.482178 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:35.482181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:35.482192 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:03:35.482234 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:35.482236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:35.482271 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:35.482341 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:35.482612 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.483353 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.484656 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.485247 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.485337 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.485404 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:35.485512 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 04:03:35.485588 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":59,"threads_started":1}
I20260924 04:03:35.485704 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.481026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41799 (local address 127.0.0.1:56452)
0924 04:03:35.481134 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.481149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:35.481234 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:35.482367 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:35.482370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:35.482386 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:35.482492 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.482498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.483443 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.483446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.484475 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.484485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.485336 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.485339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.485413 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.485435 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:03:35.485498 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:03:35.485568 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:03:35.486337 7686 tablet_service.cc:1467] Processing CreateTablet for tablet e195103f65844b86a330ffa27cf60f19 (DEFAULT_TABLE table=test-table [id=b663e48b36784e6aab860e97bf9bb196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:35.486507 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.481590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45117 (local address 127.0.0.1:36372)
0924 04:03:35.481916 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.481929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:35.482010 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:35.483078 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:35.483081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:35.483098 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:35.483170 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.483175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.484069 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.484072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.485045 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.485053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.486165 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.486168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.486231 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.486247 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:03:35.486310 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:03:35.486373 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260924 04:03:35.486532 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.481881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.0.98.2:45117)
0924 04:03:35.482885 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:35.482889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:35.482898 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:03:35.482940 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:35.482942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:35.482986 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:35.483065 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:35.483264 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.483968 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.485151 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.485750 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.485976 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.486039 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:35.486342 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 04:03:35.486413 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":79,"threads_started":1}
I20260924 04:03:35.486624 7686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e195103f65844b86a330ffa27cf60f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:35.487036 7550 tablet_service.cc:1467] Processing CreateTablet for tablet e195103f65844b86a330ffa27cf60f19 (DEFAULT_TABLE table=test-table [id=b663e48b36784e6aab860e97bf9bb196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:35.487237 7550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e195103f65844b86a330ffa27cf60f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:35.489144 7858 tablet_bootstrap.cc:492] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: Bootstrap starting.
I20260924 04:03:35.489229 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.481263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46191 (local address 127.0.0.1:39152)
0924 04:03:35.481499 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.481513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:35.481611 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:35.482751 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:35.482754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:35.482773 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:35.482858 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.482864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.483825 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.483828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.485142 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.485151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.488909 (+ 3758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.488912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.488985 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.489003 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:03:35.489068 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:03:35.489125 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":24,"thread_start_us":96,"threads_started":1}
I20260924 04:03:35.489655 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.481590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.98.1:46191)
0924 04:03:35.482561 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:35.482565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:35.482578 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:35.482622 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:35.482624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:35.482659 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:35.482718 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:35.482969 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.483715 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.485250 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.485890 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.486710 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.486773 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:35.489435 (+ 2662us) server_negotiation.cc:299] Negotiation successful
0924 04:03:35.489528 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":63,"threads_started":1}
I20260924 04:03:35.490087 7858 tablet_bootstrap.cc:654] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:35.490094 7859 tablet_bootstrap.cc:492] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: Bootstrap starting.
I20260924 04:03:35.490110 7413 tablet_service.cc:1467] Processing CreateTablet for tablet e195103f65844b86a330ffa27cf60f19 (DEFAULT_TABLE table=test-table [id=b663e48b36784e6aab860e97bf9bb196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:35.490350 7413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e195103f65844b86a330ffa27cf60f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:35.490913 7859 tablet_bootstrap.cc:654] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:35.491355 7858 tablet_bootstrap.cc:492] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: No bootstrap required, opened a new log
I20260924 04:03:35.491430 7858 ts_tablet_manager.cc:1397] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:35.491852 7858 raft_consensus.cc:348] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.491989 7858 raft_consensus.cc:374] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:35.492022 7858 raft_consensus.cc:729] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4adba512c0ac497b88071971d6c71848, State: Initialized, Role: FOLLOWER
I20260924 04:03:35.492190 7859 tablet_bootstrap.cc:492] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: No bootstrap required, opened a new log
I20260924 04:03:35.492159 7858 consensus_queue.cc:260] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.492259 7859 ts_tablet_manager.cc:1397] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:35.492494 7858 ts_tablet_manager.cc:1428] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:35.493285 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.492908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:45511)
0924 04:03:35.493134 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.493192 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260924 04:03:35.493891 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.493674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:56581)
0924 04:03:35.493770 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.493824 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 04:03:35.494823 7859 raft_consensus.cc:348] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.494900 7752 heartbeater.cc:503] Master 127.0.98.60:37355 was elected leader, sending a full tablet report...
I20260924 04:03:35.494942 7859 raft_consensus.cc:374] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:35.494974 7859 raft_consensus.cc:729] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ddabd7fc4244e1aaee11067659f6d73, State: Initialized, Role: FOLLOWER
I20260924 04:03:35.495157 7859 consensus_queue.cc:260] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.495826 7859 ts_tablet_manager.cc:1428] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260924 04:03:35.496755 7863 tablet_bootstrap.cc:492] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127: Bootstrap starting.
I20260924 04:03:35.496893 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.496382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:55393)
0924 04:03:35.496749 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.496802 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":106,"threads_started":1}
I20260924 04:03:35.497267 7616 heartbeater.cc:503] Master 127.0.98.60:37355 was elected leader, sending a full tablet report...
I20260924 04:03:35.497519 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.497304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:49383)
0924 04:03:35.497436 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.497472 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 04:03:35.498382 7863 tablet_bootstrap.cc:654] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:35.499532 7863 tablet_bootstrap.cc:492] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127: No bootstrap required, opened a new log
I20260924 04:03:35.499617 7863 ts_tablet_manager.cc:1397] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:35.500056 7863 raft_consensus.cc:348] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.500145 7863 raft_consensus.cc:374] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:35.500174 7863 raft_consensus.cc:729] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c19b464c064f27bab5b704f1ff6127, State: Initialized, Role: FOLLOWER
I20260924 04:03:35.500293 7863 consensus_queue.cc:260] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.500568 7863 ts_tablet_manager.cc:1428] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:35.501549 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.500840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:57365)
0924 04:03:35.501086 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.501457 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260924 04:03:35.502100 7479 heartbeater.cc:503] Master 127.0.98.60:37355 was elected leader, sending a full tablet report...
I20260924 04:03:35.502590 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.501966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:47085)
0924 04:03:35.502443 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.502507 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":96,"threads_started":1}
I20260924 04:03:35.528445 7865 raft_consensus.cc:482] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:35.528604 7865 raft_consensus.cc:504] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.529654 7865 leader_election.cc:288] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4adba512c0ac497b88071971d6c71848 (127.0.98.3:41799), f7c19b464c064f27bab5b704f1ff6127 (127.0.98.1:46191)
W20260924 04:03:35.532403 7755 tablet.cc:2369] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:35.534066 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.529937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60017 (local address 127.0.98.1:46191)
0924 04:03:35.530076 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:35.530081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:35.530097 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:03:35.530223 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:35.530226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:35.530280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:35.530341 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:35.530619 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.531417 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.532871 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.533476 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.533559 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.533609 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:35.533875 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 04:03:35.533947 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:03:35.534065 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.529957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44115 (local address 127.0.98.3:41799)
0924 04:03:35.530076 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:35.530080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:35.530415 (+ 335us) server_negotiation.cc:407] Connection header received
0924 04:03:35.530494 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:35.530496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:35.530536 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:35.530594 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:35.530798 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.531526 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.532785 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.533406 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.533526 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.533590 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:35.533875 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 04:03:35.533945 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:03:35.534068 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.529848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46191 (local address 127.0.98.2:60017)
0924 04:03:35.529975 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.529989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:35.530121 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:35.530372 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:35.530374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:35.530395 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:35.530529 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.530534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.531602 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.531606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.532763 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.532773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.533562 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.533566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.533737 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.533778 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:03:35.533843 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:03:35.533912 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":82,"spinlock_wait_cycles":384}
I20260924 04:03:35.534148 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.529848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41799 (local address 127.0.98.2:44115)
0924 04:03:35.530323 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.530335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:35.530419 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:35.530618 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:35.530620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:35.530636 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:35.530713 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.530717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.531622 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.531625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.532685 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:35.532697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.533529 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.533532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:35.533732 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.533774 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 04:03:35.533870 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 04:03:35.533917 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":232,"thread_start_us":127,"threads_started":1}
I20260924 04:03:35.534495 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e195103f65844b86a330ffa27cf60f19" candidate_uuid: "7ddabd7fc4244e1aaee11067659f6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4adba512c0ac497b88071971d6c71848" is_pre_election: true
I20260924 04:03:35.534518 7433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e195103f65844b86a330ffa27cf60f19" candidate_uuid: "7ddabd7fc4244e1aaee11067659f6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c19b464c064f27bab5b704f1ff6127" is_pre_election: true
I20260924 04:03:35.534665 7706 raft_consensus.cc:2454] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ddabd7fc4244e1aaee11067659f6d73 in term 0.
I20260924 04:03:35.534714 7433 raft_consensus.cc:2454] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ddabd7fc4244e1aaee11067659f6d73 in term 0.
I20260924 04:03:35.535068 7505 leader_election.cc:302] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4adba512c0ac497b88071971d6c71848, 7ddabd7fc4244e1aaee11067659f6d73; no voters:
I20260924 04:03:35.535301 7865 raft_consensus.cc:2781] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:35.535375 7865 raft_consensus.cc:482] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:35.535426 7865 raft_consensus.cc:3037] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:35.536414 7865 raft_consensus.cc:504] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.536697 7865 leader_election.cc:288] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [CANDIDATE]: Term 1 election: Requested vote from peers 4adba512c0ac497b88071971d6c71848 (127.0.98.3:41799), f7c19b464c064f27bab5b704f1ff6127 (127.0.98.1:46191)
I20260924 04:03:35.537003 7433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e195103f65844b86a330ffa27cf60f19" candidate_uuid: "7ddabd7fc4244e1aaee11067659f6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c19b464c064f27bab5b704f1ff6127"
I20260924 04:03:35.537003 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e195103f65844b86a330ffa27cf60f19" candidate_uuid: "7ddabd7fc4244e1aaee11067659f6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4adba512c0ac497b88071971d6c71848"
I20260924 04:03:35.537115 7706 raft_consensus.cc:3037] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:35.537115 7433 raft_consensus.cc:3037] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:35.538228 7706 raft_consensus.cc:2454] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ddabd7fc4244e1aaee11067659f6d73 in term 1.
I20260924 04:03:35.538300 7433 raft_consensus.cc:2454] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ddabd7fc4244e1aaee11067659f6d73 in term 1.
I20260924 04:03:35.538487 7505 leader_election.cc:302] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4adba512c0ac497b88071971d6c71848, 7ddabd7fc4244e1aaee11067659f6d73; no voters:
I20260924 04:03:35.538646 7865 raft_consensus.cc:2781] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:35.538887 7865 raft_consensus.cc:686] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 LEADER]: Becoming Leader. State: Replica: 7ddabd7fc4244e1aaee11067659f6d73, State: Running, Role: LEADER
I20260924 04:03:35.539155 7865 consensus_queue.cc:237] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } }
I20260924 04:03:35.539520 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.539321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:45953)
0924 04:03:35.539409 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.539459 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 04:03:35.541503 7300 catalog_manager.cc:5495] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ddabd7fc4244e1aaee11067659f6d73 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7ddabd7fc4244e1aaee11067659f6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ddabd7fc4244e1aaee11067659f6d73" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45117 } health_report { overall_health: HEALTHY } } }
I20260924 04:03:35.563968 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.560559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.0.98.2:45117)
0924 04:03:35.560677 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:35.560682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:35.560695 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:35.560805 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:35.560808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:35.560868 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:35.560956 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:35.561202 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.561705 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.562526 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:35.562752 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:35.562837 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:35.563091 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:35.563559 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:35.563660 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:35.563733 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 04:03:35.563791 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:03:35.566773 7706 raft_consensus.cc:1261] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 1 FOLLOWER]: Refusing update from remote peer 7ddabd7fc4244e1aaee11067659f6d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:35.566934 7433 raft_consensus.cc:1261] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 1 FOLLOWER]: Refusing update from remote peer 7ddabd7fc4244e1aaee11067659f6d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:35.567328 7872 consensus_queue.cc:1035] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4adba512c0ac497b88071971d6c71848" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:35.567514 7865 consensus_queue.cc:1035] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c19b464c064f27bab5b704f1ff6127" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:03:35.569628 7619 tablet.cc:2369] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:35.571067 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.570803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:45549)
0924 04:03:35.570942 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.571002 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 04:03:35.572961 7875 mvcc.cc:205] Tried to move back new op lower bound from 7332751833356619776 to 7332751833251127296. Current Snapshot: MvccSnapshot[applied={T|T < 7332751833356619776}]
I20260924 04:03:35.574424 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.574173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:47533)
0924 04:03:35.574265 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.574350 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 04:03:35.574678 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.571752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:41495)
0924 04:03:35.574540 (+ 2788us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.574601 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2713}
I20260924 04:03:35.575825 7877 mvcc.cc:205] Tried to move back new op lower bound from 7332751833356619776 to 7332751833251127296. Current Snapshot: MvccSnapshot[applied={T|T < 7332751833356619776}]
I20260924 04:03:35.577215 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.576224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.1:41415)
0924 04:03:35.577092 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.577147 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":797}
I20260924 04:03:35.577664 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.577002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.2:36363)
0924 04:03:35.577550 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.577600 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485}
I20260924 04:03:35.580513 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.580226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46295 (local address 127.0.98.3:49253)
0924 04:03:35.580394 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.580441 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 04:03:35.582737 7299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:35.582865 7299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:35.587050 7299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc33482acaf44742a3028d36fbfd81ee: Sending DeleteTablet for 3 replicas of tablet e195103f65844b86a330ffa27cf60f19
I20260924 04:03:35.587643 7686 tablet_service.cc:1514] Processing DeleteTablet for tablet e195103f65844b86a330ffa27cf60f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:35 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56452
I20260924 04:03:35.588024 7885 tablet_replica.cc:330] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: stopping tablet replica
I20260924 04:03:35.587999 7413 tablet_service.cc:1514] Processing DeleteTablet for tablet e195103f65844b86a330ffa27cf60f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:35 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39152
I20260924 04:03:35.588042 7550 tablet_service.cc:1514] Processing DeleteTablet for tablet e195103f65844b86a330ffa27cf60f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:35 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36372
I20260924 04:03:35.588342 7886 tablet_replica.cc:330] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127: stopping tablet replica
I20260924 04:03:35.588387 7885 raft_consensus.cc:2229] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:35.588469 7886 raft_consensus.cc:2229] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:35.588608 7886 raft_consensus.cc:2258] T e195103f65844b86a330ffa27cf60f19 P f7c19b464c064f27bab5b704f1ff6127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:35.588696 7887 tablet_replica.cc:330] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: stopping tablet replica
I20260924 04:03:35.588799 7887 raft_consensus.cc:2229] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:35.589115 7887 raft_consensus.cc:2258] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:35.589207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 7260
I20260924 04:03:35.590190 7887 ts_tablet_manager.cc:1905] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:35.592720 7887 ts_tablet_manager.cc:1918] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:35.592783 7887 log.cc:1196] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/e195103f65844b86a330ffa27cf60f19
I20260924 04:03:35.593058 7887 ts_tablet_manager.cc:1939] T e195103f65844b86a330ffa27cf60f19 P 7ddabd7fc4244e1aaee11067659f6d73: Deleting consensus metadata
I20260924 04:03:35.593894 7287 catalog_manager.cc:4841] TS 7ddabd7fc4244e1aaee11067659f6d73 (127.0.98.2:45117): tablet e195103f65844b86a330ffa27cf60f19 (table test-table [id=b663e48b36784e6aab860e97bf9bb196]) successfully deleted
I20260924 04:03:35.596602 7885 raft_consensus.cc:2258] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:35.597333 7885 ts_tablet_manager.cc:1905] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:35.598825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 7487
I20260924 04:03:35.599094 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.598772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46191 (local address 127.0.0.1:39166)
0924 04:03:35.598939 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.598989 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260924 04:03:35.600368 7287 catalog_manager.cc:4584] TS f7c19b464c064f27bab5b704f1ff6127 (127.0.98.1:46191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e195103f65844b86a330ffa27cf60f19: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46191: connect: Connection refused (error 111)
I20260924 04:03:35.603986 7885 ts_tablet_manager.cc:1918] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:35.604081 7885 log.cc:1196] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/e195103f65844b86a330ffa27cf60f19
I20260924 04:03:35.604120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 7623
I20260924 04:03:35.604364 7885 ts_tablet_manager.cc:1939] T e195103f65844b86a330ffa27cf60f19 P 4adba512c0ac497b88071971d6c71848: Deleting consensus metadata
I20260924 04:03:35.609711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 6979
I20260924 04:03:35.610383 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:35.609886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41799 (local address 127.0.0.1:56462)
0924 04:03:35.610254 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:35.610303 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:41799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260924 04:03:35.614754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 7083
2026-09-24T04:03:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7071 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:35 dist-test-slave-zcsk krb5kdc[7890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:35 dist-test-slave-zcsk krb5kdc[7890](info): set up 2 sockets
Sep 24 04:03:35 dist-test-slave-zcsk krb5kdc[7890](info): commencing operation
krb5kdc: starting...
W20260924 04:03:36.580265 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:36 dist-test-slave-zcsk krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:03:36.699288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:36.807191 7903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:36.807489 7903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:36.807545 7903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:36.811581 7903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:36.811658 7903 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:03:36.811681 7903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:36.811703 7903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:36.811723 7903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:36.811748 7903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:36.817443 7903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43883
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:36.818535 7903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:36.819572 7903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:36.828477 7908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:36.828548 7909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:36.828543 7903 server_base.cc:1034] running on GCE node
W20260924 04:03:36.828454 7911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:36.829275 7903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:36.830418 7903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:36.831573 7903 hybrid_clock.cc:648] HybridClock initialized: now 1790222616831546 us; error 43 us; skew 500 ppm
Sep 24 04:03:36 dist-test-slave-zcsk krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:36.834331 7903 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:03:36.835263 7903 webserver.cc:460] Webserver started at http://127.0.98.62:40573/ using document root <none> and password file <none>
I20260924 04:03:36.835816 7903 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:36.835889 7903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:36.836133 7903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:36.838191 7903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "fcb596637a1049dbb101c01c7d75bf25"
format_stamp: "Formatted at 2026-09-24 04:03:36 on dist-test-slave-zcsk"
server_key: "250845e5029bdd1b7b9cb034fd347ad8"
server_key_iv: "d5a67bd8c11d1ce433c640d9fe9db1fc"
server_key_version: "encryptionkey@0"
I20260924 04:03:36.838770 7903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "fcb596637a1049dbb101c01c7d75bf25"
format_stamp: "Formatted at 2026-09-24 04:03:36 on dist-test-slave-zcsk"
server_key: "250845e5029bdd1b7b9cb034fd347ad8"
server_key_iv: "d5a67bd8c11d1ce433c640d9fe9db1fc"
server_key_version: "encryptionkey@0"
I20260924 04:03:36.842744 7903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:03:36.845331 7918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:36.846626 7903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:36.846750 7903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "fcb596637a1049dbb101c01c7d75bf25"
format_stamp: "Formatted at 2026-09-24 04:03:36 on dist-test-slave-zcsk"
server_key: "250845e5029bdd1b7b9cb034fd347ad8"
server_key_iv: "d5a67bd8c11d1ce433c640d9fe9db1fc"
server_key_version: "encryptionkey@0"
I20260924 04:03:36.846849 7903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:36.871209 7903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:36.874477 7903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:36.874660 7903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:36.882586 7903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43883
I20260924 04:03:36.882606 7980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43883 every 8 connection(s)
I20260924 04:03:36.883630 7903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:36.885425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 7903
I20260924 04:03:36.885545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:36.885798 392 external_mini_cluster.cc:1428] Setting key 0f226fcf28b1f73151b69a1ed71e50f2
I20260924 04:03:36.886734 7981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:36.893591 7981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: Bootstrap starting.
Sep 24 04:03:36 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:36.896260 7981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:36.897305 7981 log.cc:824] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:36.899555 7981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: No bootstrap required, opened a new log
I20260924 04:03:36.900996 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:36.887130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.0.98.62:43883)
0924 04:03:36.887527 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:36.887545 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:36.887586 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:03:36.888464 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:36.888488 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:36.888937 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:36.889321 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:36.889996 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:36.891437 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:36.892428 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:36.893121 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:36.893281 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:36.896222 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:36.896228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:36.896242 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:36.896272 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:36.898005 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:36.898495 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:36.898500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:36.898505 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:36.898623 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:36.898949 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:36.898951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:36.898952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:36.899381 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:36.899525 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:36.899868 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 04:03:36.900216 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
I20260924 04:03:36.903007 7981 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb596637a1049dbb101c01c7d75bf25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43883 } }
I20260924 04:03:36.903288 7981 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:36.903349 7981 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb596637a1049dbb101c01c7d75bf25, State: Initialized, Role: FOLLOWER
I20260924 04:03:36.903796 7981 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb596637a1049dbb101c01c7d75bf25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43883 } }
I20260924 04:03:36.903952 7981 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:36.904021 7981 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:36.904119 7981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:36.905031 7981 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb596637a1049dbb101c01c7d75bf25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43883 } }
I20260924 04:03:36.905371 7981 leader_election.cc:302] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcb596637a1049dbb101c01c7d75bf25; no voters:
I20260924 04:03:36.905680 7981 leader_election.cc:288] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:36.905804 7986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:36.906028 7986 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [term 1 LEADER]: Becoming Leader. State: Replica: fcb596637a1049dbb101c01c7d75bf25, State: Running, Role: LEADER
I20260924 04:03:36.906338 7986 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb596637a1049dbb101c01c7d75bf25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43883 } }
I20260924 04:03:36.906863 7981 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:36.908056 7987 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcb596637a1049dbb101c01c7d75bf25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb596637a1049dbb101c01c7d75bf25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43883 } } }
I20260924 04:03:36.908054 7988 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb596637a1049dbb101c01c7d75bf25. Latest consensus state: current_term: 1 leader_uuid: "fcb596637a1049dbb101c01c7d75bf25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb596637a1049dbb101c01c7d75bf25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43883 } } }
I20260924 04:03:36.908201 7988 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:36.908201 7987 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:36.908607 7996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:36.911571 7996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:36.917169 7996 catalog_manager.cc:1347] Generated new cluster ID: 88618aad7f9a407e82c51a592c0cb03c
I20260924 04:03:36.917253 7996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:36.929854 7996 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:36.930814 7996 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:36.935074 7996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: Generated new TSK 0
I20260924 04:03:36.935771 7996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:03:36.971494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43883
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:37.084487 8007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:37.084779 8007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:37.084824 8007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:37.089012 8007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:37.089097 8007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:37.089138 8007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:37.089170 8007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:37.094774 8007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:37.095990 8007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:37.097096 8007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:37.106703 8012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:37.106774 8015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:37.106696 8013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:37.106832 8007 server_base.cc:1034] running on GCE node
I20260924 04:03:37.107390 8007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:37.108096 8007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:37.109268 8007 hybrid_clock.cc:648] HybridClock initialized: now 1790222617109239 us; error 45 us; skew 500 ppm
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:37.112037 8007 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:03:37.112975 8007 webserver.cc:460] Webserver started at http://127.0.98.1:40719/ using document root <none> and password file <none>
I20260924 04:03:37.113550 8007 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:37.113620 8007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:37.113839 8007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:37.115985 8007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "c197519593bece261fa2780e1908e5d3"
server_key_iv: "36f2efb1eae55d2d1a7221a8f6263d60"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.116554 8007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "c197519593bece261fa2780e1908e5d3"
server_key_iv: "36f2efb1eae55d2d1a7221a8f6263d60"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.120383 8007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:03:37.122776 8022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:37.123986 8007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:37.124095 8007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "c197519593bece261fa2780e1908e5d3"
server_key_iv: "36f2efb1eae55d2d1a7221a8f6263d60"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.124202 8007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:37.156598 8007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:37.160151 8007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:37.160338 8007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:37.161845 8007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:37.161899 8007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:37.161937 8007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:37.161970 8007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:37.172210 8007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39811
I20260924 04:03:37.172252 8137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39811 every 8 connection(s)
I20260924 04:03:37.173113 8007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:37.175951 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.162821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42475 (local address 127.0.98.62:43883)
0924 04:03:37.162944 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.162948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.164093 (+ 1145us) server_negotiation.cc:407] Connection header received
0924 04:03:37.165133 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.165136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.165250 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.165334 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:37.166623 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.167153 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.168972 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.169301 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.169407 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.172346 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:37.172351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:37.172353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:37.172384 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:37.173979 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.174462 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.174465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.174467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.174547 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.174863 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.174867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.174869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.175053 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:37.175135 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.175618 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.175747 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:03:37.176728 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.163373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43883 (local address 127.0.98.1:42475)
0924 04:03:37.163942 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.163994 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.164936 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.165590 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.165597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.165925 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:37.166483 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.166497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.167305 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.167309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.168830 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.168842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.169469 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.169473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.169666 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.170297 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:37.170320 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:37.172146 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:37.174099 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.174105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.174119 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.174357 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.174631 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.174634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.174636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.174770 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.175146 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:37.175153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:37.175391 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.175548 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.175878 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":136,"threads_started":1}
I20260924 04:03:37.178023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 8007
I20260924 04:03:37.178020 8138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43883
I20260924 04:03:37.178150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:37.178172 8138 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:37.178509 392 external_mini_cluster.cc:1428] Setting key ebbd7bbfb994e40c358852243322cff9
I20260924 04:03:37.178741 8138 heartbeater.cc:511] Master 127.0.98.62:43883 requested a full tablet report, sending...
I20260924 04:03:37.180385 7935 ts_manager.cc:194] Registered new tserver with Master: e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811)
I20260924 04:03:37.181833 7935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42475
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:03:37.188470 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.180186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54477 (local address 127.0.98.62:43883)
0924 04:03:37.180296 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.180299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.182299 (+ 2000us) server_negotiation.cc:407] Connection header received
0924 04:03:37.182412 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.182416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.182469 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.182556 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:37.183017 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.183511 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.184341 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.184557 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.184638 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.185631 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:37.185634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:37.185636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:37.185656 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:37.186958 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.187333 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.187337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.187339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.187418 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.187656 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.187659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.187661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.187827 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:37.187938 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.188192 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.188305 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:03:37.188480 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.181879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43883 (local address 127.0.98.1:54477)
0924 04:03:37.182203 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.182217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.182317 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.182588 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.182591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.182787 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:37.182905 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.182911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.183629 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.183632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.184239 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.184248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.184650 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.184654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.184736 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.185190 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:37.185201 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:37.185507 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:37.187058 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.187061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.187063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.187249 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.187498 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.187501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.187503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.187580 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.187938 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:37.187941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:37.188033 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.188131 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.188289 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:03:37.206482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43883
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:37.315402 8147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:37.315690 8147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:37.315737 8147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:37.319794 8147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:37.319860 8147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:37.319900 8147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:37.319950 8147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:37.325452 8147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:37.326421 8147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:37.327332 8147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:37.336381 8155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:37.336419 8153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:37.336442 8152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:37.336557 8147 server_base.cc:1034] running on GCE node
I20260924 04:03:37.337035 8147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:37.337641 8147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:37.338789 8147 hybrid_clock.cc:648] HybridClock initialized: now 1790222617338765 us; error 37 us; skew 500 ppm
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:37.341297 8147 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:03:37.342093 8147 webserver.cc:460] Webserver started at http://127.0.98.2:35155/ using document root <none> and password file <none>
I20260924 04:03:37.342598 8147 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:37.342661 8147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:37.342875 8147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:37.344897 8147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "2d09b566136849aaa498bb26586f4fa9"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "46247a82c1f8026579986fbd4421319b"
server_key_iv: "c8734778562351581f3e855b764ecbef"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.345412 8147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2d09b566136849aaa498bb26586f4fa9"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "46247a82c1f8026579986fbd4421319b"
server_key_iv: "c8734778562351581f3e855b764ecbef"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.348937 8147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:37.351312 8162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:37.352547 8147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:37.352648 8147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "2d09b566136849aaa498bb26586f4fa9"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "46247a82c1f8026579986fbd4421319b"
server_key_iv: "c8734778562351581f3e855b764ecbef"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.352741 8147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:37.365128 8147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:37.367870 8147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:37.368064 8147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:37.369455 8147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:37.369503 8147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:37.369555 8147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:37.369585 8147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:37.379350 8147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33641
I20260924 04:03:37.379426 8277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33641 every 8 connection(s)
I20260924 04:03:37.380429 8147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:37.382453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 8147
I20260924 04:03:37.382577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:37.382889 392 external_mini_cluster.cc:1428] Setting key 6c0e50a8ebd2284f53b245976e0b1bb1
I20260924 04:03:37.383066 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.370317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51173 (local address 127.0.98.62:43883)
0924 04:03:37.370433 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.370437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.371467 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 04:03:37.372587 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.372592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.372651 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.372708 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:37.374421 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.375249 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.376838 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.377058 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.377146 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.379726 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:37.379730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:37.379733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:37.379762 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:37.381252 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.381677 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.381679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.381681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.381750 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.381999 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.382001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.382002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.382157 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:37.382227 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.382794 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.382914 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:03:37.384014 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.370857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43883 (local address 127.0.98.2:51173)
0924 04:03:37.371319 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.371360 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.372325 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.373149 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.373158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.373569 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:37.374242 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.374261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.375413 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.375416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.376711 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.376721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.377195 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.377199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.377397 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.378008 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:37.378027 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:37.379550 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:37.381350 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.381355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.381369 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.381600 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.381813 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.381815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.381816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.381937 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.382253 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:37.382260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:37.382562 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.382786 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.383106 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20260924 04:03:37.385375 8278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43883
I20260924 04:03:37.385515 8278 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:37.386066 8278 heartbeater.cc:511] Master 127.0.98.62:43883 requested a full tablet report, sending...
I20260924 04:03:37.387176 7934 ts_manager.cc:194] Registered new tserver with Master: 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641)
I20260924 04:03:37.387940 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51173
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 04:03:37.394948 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.387470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55595 (local address 127.0.98.62:43883)
0924 04:03:37.387566 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.387570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.387798 (+ 228us) server_negotiation.cc:407] Connection header received
0924 04:03:37.387899 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.387936 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.387982 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.388054 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:37.388611 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.389334 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.390479 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.390685 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.390765 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.391980 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:37.391984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:37.391987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:37.392012 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:37.393349 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.393754 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.393758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.393760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.393835 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.394068 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.394071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.394073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.394348 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:37.394434 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.394692 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.394792 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:03:37.394970 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.387408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43883 (local address 127.0.98.2:55595)
0924 04:03:37.387705 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.387718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.387820 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.388154 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.388157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.388374 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:37.388491 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.388499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.389475 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.389479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.390364 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.390374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.390786 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.390790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.390883 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.391404 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:37.391418 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:37.391818 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:37.393480 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.393483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.393485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.393671 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.393912 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.393915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.393917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.393993 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.394440 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:37.394443 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:37.394530 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.394611 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.394761 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:03:37.411701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43883
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:37.520298 8287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:37.520572 8287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:37.520619 8287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:37.524660 8287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:37.524722 8287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:37.524765 8287 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:03:37.524817 8287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:37.530242 8287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:37.531315 8287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:37.532382 8287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:37.542111 8292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:37.542233 8287 server_base.cc:1034] running on GCE node
W20260924 04:03:37.542127 8293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:37.542111 8295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:37.542801 8287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:37.543478 8287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:37.544620 8287 hybrid_clock.cc:648] HybridClock initialized: now 1790222617544591 us; error 58 us; skew 500 ppm
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:37.547235 8287 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:03:37.548125 8287 webserver.cc:460] Webserver started at http://127.0.98.3:44095/ using document root <none> and password file <none>
I20260924 04:03:37.548645 8287 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:37.548709 8287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:37.548898 8287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:37.551033 8287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a621eebd4414b4da0d5dac00f35bee0"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "c502687c18901fcc508a91f32e3bdf16"
server_key_iv: "81b26a5df25b65f782f1b648191ace17"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.551563 8287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a621eebd4414b4da0d5dac00f35bee0"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "c502687c18901fcc508a91f32e3bdf16"
server_key_iv: "81b26a5df25b65f782f1b648191ace17"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.555160 8287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:03:37.557699 8302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:37.558944 8287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:37.559051 8287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "1a621eebd4414b4da0d5dac00f35bee0"
format_stamp: "Formatted at 2026-09-24 04:03:37 on dist-test-slave-zcsk"
server_key: "c502687c18901fcc508a91f32e3bdf16"
server_key_iv: "81b26a5df25b65f782f1b648191ace17"
server_key_version: "encryptionkey@0"
I20260924 04:03:37.559147 8287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:37.593601 8287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:37.597067 8287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:37.597278 8287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:37.598839 8287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:37.598910 8287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:37.598958 8287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:37.598992 8287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:37.609741 8287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40113
I20260924 04:03:37.610283 8417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40113 every 8 connection(s)
I20260924 04:03:37.610841 8287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:37.612689 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.599726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50273 (local address 127.0.98.62:43883)
0924 04:03:37.599838 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.599841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.600939 (+ 1098us) server_negotiation.cc:407] Connection header received
0924 04:03:37.601972 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.601977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.602038 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.602094 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:37.603786 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.604604 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.605805 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.606036 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.606125 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.608990 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:37.608993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:37.608996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:37.609022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:37.610566 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.611168 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.611171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.611172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.611259 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.611668 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.611672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.611673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.611850 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:37.611962 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.612467 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.612570 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:03:37.613435 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.600297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43883 (local address 127.0.98.3:50273)
0924 04:03:37.600786 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.600830 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.601721 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.602536 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.602546 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.602943 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:37.603593 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.603615 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.604765 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.604769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.605683 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.605693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.606196 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.606200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.606367 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.606995 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:37.607012 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:37.608789 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:37.610672 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.610678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.610693 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.611033 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.611374 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.611377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.611379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.611555 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.611966 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:37.611973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:37.612217 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.612387 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.612642 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260924 04:03:37.614725 8418 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43883
I20260924 04:03:37.614864 8418 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:37.615330 8418 heartbeater.cc:511] Master 127.0.98.62:43883 requested a full tablet report, sending...
I20260924 04:03:37.616371 7935 ts_manager.cc:194] Registered new tserver with Master: 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113)
I20260924 04:03:37.617067 7935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50273
I20260924 04:03:37.617820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 8287
I20260924 04:03:37.617937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:37.618206 392 external_mini_cluster.cc:1428] Setting key ef28425632ba35e67aa0bbd90411f53c
I20260924 04:03:37.620002 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:03:37.623850 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.616727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43883 (local address 127.0.98.3:37919)
0924 04:03:37.616980 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.616991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.617083 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.617342 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.617344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.617520 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:37.617660 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.617667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.618689 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.618692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.619487 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.619497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.620013 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.620016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.620104 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.620571 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:37.620585 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:37.620894 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:37.622627 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.622630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.622631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.622820 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.623062 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:37.623064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:37.623066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:37.623131 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:37.623470 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:37.623472 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:37.623543 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.623613 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.623719 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
I20260924 04:03:37.623950 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.616779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37919 (local address 127.0.98.62:43883)
0924 04:03:37.616883 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.616887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.617056 (+ 169us) server_negotiation.cc:407] Connection header received
0924 04:03:37.617201 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.617204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.617250 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.617314 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:37.617781 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.618519 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.619662 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.619920 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.620011 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.621020 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:37.621023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:37.621026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:37.621050 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:37.622509 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.622899 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.622902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.622904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.622985 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.623200 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.623203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.623205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.623390 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:37.623470 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.623670 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.623776 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:03:37.643568 7984 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:37.637725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.0.98.62:43883)
0924 04:03:37.637867 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.637871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.637884 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:37.637924 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.637927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.637965 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.638038 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:37.638703 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.639232 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.640099 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.640323 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.640414 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.643440 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44484: BlockingRecv error: recv got EOF from 127.0.0.1:44484 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 04:03:37 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222617, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:37.652679 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.645137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.0.98.62:43883)
0924 04:03:37.645267 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.645272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.645367 (+ 95us) server_negotiation.cc:407] Connection header received
0924 04:03:37.645495 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.645497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.645537 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.645599 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:37.646188 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.646691 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.647458 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.647682 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.647768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.649951 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:37.649955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:37.649957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:37.649982 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:37.651326 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.651681 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.651684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.651685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.651757 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:37.651986 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:37.651988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:37.651989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:37.652151 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:37.652246 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.652429 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.652523 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:03:37.655426 7935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:37.658519 7935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:37.672588 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.667741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.0.98.3:40113)
0924 04:03:37.668073 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.668083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.668100 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:03:37.668156 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.668160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.668358 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.668516 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:37.669039 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.670177 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.671305 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.671934 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.672032 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.672198 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.672336 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.672430 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260924 04:03:37.672752 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.666669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39811 (local address 127.0.0.1:46222)
0924 04:03:37.667542 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.667598 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.667730 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.668519 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.668526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.668548 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:37.668690 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.668706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.670235 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.670239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.671422 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.671430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.672390 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.672393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.672470 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.672489 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.672542 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.672603 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":46,"spinlock_wait_cycles":2816,"thread_start_us":54,"threads_started":1}
I20260924 04:03:37.672808 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.666848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.0.98.1:39811)
0924 04:03:37.668021 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.668036 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.668059 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:03:37.668139 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.668144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.668364 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.668516 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:37.668850 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.670076 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.671534 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.672302 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.672392 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.672539 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.672585 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.672680 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044,"thread_start_us":78,"threads_started":1}
I20260924 04:03:37.673049 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.666937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40113 (local address 127.0.0.1:56180)
0924 04:03:37.667939 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.667954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.668032 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.668819 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.668822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.668838 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:37.668915 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.668920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.670285 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.670288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.671187 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.671198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.672046 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.672049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.672150 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.672187 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.672289 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.672898 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":369,"thread_start_us":89,"threads_started":1}
I20260924 04:03:37.672588 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.666925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33641 (local address 127.0.0.1:55432)
0924 04:03:37.667542 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.667598 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.667730 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.668616 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.668619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.668632 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:37.668712 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.668717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.669793 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.669797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.671080 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.671091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.672077 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.672081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.672195 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.672221 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.672273 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.672431 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":2,"spinlock_wait_cycles":5632,"thread_start_us":116,"threads_started":1}
I20260924 04:03:37.674055 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.667406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.0.98.2:33641)
0924 04:03:37.668241 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.668250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.668268 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:03:37.668326 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.668329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.668482 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.668851 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:37.668973 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.669691 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.671204 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.671945 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.673629 (+ 1684us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.673791 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.673839 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.673924 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":96,"threads_started":1}
I20260924 04:03:37.675536 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 40a5b29c72414818a6b56a84b7ac6f86 (DEFAULT_TABLE table=test-table [id=c76401badd794ede837a817385b53b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:37.675534 8072 tablet_service.cc:1467] Processing CreateTablet for tablet 40a5b29c72414818a6b56a84b7ac6f86 (DEFAULT_TABLE table=test-table [id=c76401badd794ede837a817385b53b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:37.676512 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a5b29c72414818a6b56a84b7ac6f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:37.676548 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a5b29c72414818a6b56a84b7ac6f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:37.678664 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 40a5b29c72414818a6b56a84b7ac6f86 (DEFAULT_TABLE table=test-table [id=c76401badd794ede837a817385b53b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:37.679684 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a5b29c72414818a6b56a84b7ac6f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:37.683485 8446 tablet_bootstrap.cc:492] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Bootstrap starting.
I20260924 04:03:37.684841 8447 tablet_bootstrap.cc:492] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: Bootstrap starting.
I20260924 04:03:37.686096 8446 tablet_bootstrap.cc:654] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:37.686942 8446 log.cc:824] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:37.687179 8448 tablet_bootstrap.cc:492] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: Bootstrap starting.
I20260924 04:03:37.687248 8447 tablet_bootstrap.cc:654] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:37.688072 8447 log.cc:824] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:37.688880 8446 tablet_bootstrap.cc:492] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: No bootstrap required, opened a new log
I20260924 04:03:37.689066 8446 ts_tablet_manager.cc:1397] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:03:37.689570 8448 tablet_bootstrap.cc:654] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:37.690068 8447 tablet_bootstrap.cc:492] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: No bootstrap required, opened a new log
I20260924 04:03:37.690177 8448 log.cc:824] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:37.690243 8447 ts_tablet_manager.cc:1397] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:03:37.691675 8448 tablet_bootstrap.cc:492] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: No bootstrap required, opened a new log
I20260924 04:03:37.691835 8448 ts_tablet_manager.cc:1397] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:03:37.692867 8446 raft_consensus.cc:348] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.693126 8446 raft_consensus.cc:374] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:37.693178 8446 raft_consensus.cc:729] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a621eebd4414b4da0d5dac00f35bee0, State: Initialized, Role: FOLLOWER
I20260924 04:03:37.693614 8446 consensus_queue.cc:260] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.693787 8447 raft_consensus.cc:348] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.694021 8447 raft_consensus.cc:374] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:37.694074 8447 raft_consensus.cc:729] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a7481f0e914a44a1c2db3a0aeffc6c, State: Initialized, Role: FOLLOWER
I20260924 04:03:37.694501 8447 consensus_queue.cc:260] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.694584 8446 ts_tablet_manager.cc:1428] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:03:37.694823 8448 raft_consensus.cc:348] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.695047 8448 raft_consensus.cc:374] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:37.695096 8448 raft_consensus.cc:729] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d09b566136849aaa498bb26586f4fa9, State: Initialized, Role: FOLLOWER
I20260924 04:03:37.695091 8138 heartbeater.cc:503] Master 127.0.98.62:43883 was elected leader, sending a full tablet report...
I20260924 04:03:37.695319 8447 ts_tablet_manager.cc:1428] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:03:37.695405 8418 heartbeater.cc:503] Master 127.0.98.62:43883 was elected leader, sending a full tablet report...
I20260924 04:03:37.696353 8448 consensus_queue.cc:260] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.696980 8278 heartbeater.cc:503] Master 127.0.98.62:43883 was elected leader, sending a full tablet report...
I20260924 04:03:37.697216 8448 ts_tablet_manager.cc:1428] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:03:37.813082 8452 raft_consensus.cc:482] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:37.813287 8452 raft_consensus.cc:504] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.814397 8452 leader_election.cc:288] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811), 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641)
I20260924 04:03:37.818879 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.814865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44245 (local address 127.0.98.2:33641)
0924 04:03:37.815001 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.815006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.815019 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:37.815070 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.815073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.815135 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.815214 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:37.815510 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.816256 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.817673 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.818346 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.818470 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.818542 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.818693 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.818763 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:03:37.818902 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.814655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33641 (local address 127.0.98.3:44245)
0924 04:03:37.814787 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.814812 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.814910 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.815260 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.815263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.815283 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:37.815396 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.815403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.816363 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.816366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.817540 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.817548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.818467 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.818470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.818573 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.818592 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.818658 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.818738 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":2}
I20260924 04:03:37.819607 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a5b29c72414818a6b56a84b7ac6f86" candidate_uuid: "1a621eebd4414b4da0d5dac00f35bee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d09b566136849aaa498bb26586f4fa9" is_pre_election: true
I20260924 04:03:37.819705 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.814911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60383 (local address 127.0.98.1:39811)
0924 04:03:37.815028 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.815032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.815423 (+ 391us) server_negotiation.cc:407] Connection header received
0924 04:03:37.815699 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.815703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.815752 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.815827 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:37.816525 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.817275 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.818696 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.819230 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.819316 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.819361 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.819562 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.819614 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:03:37.819749 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.814655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39811 (local address 127.0.98.3:60383)
0924 04:03:37.815330 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:37.815341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:37.815428 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:37.815842 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:37.815845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:37.815867 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:37.816419 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.816427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.817417 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.817420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.818575 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:37.818584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.819359 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.819362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:37.819449 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.819470 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:03:37.819540 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:03:37.819610 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":352,"thread_start_us":116,"threads_started":1}
I20260924 04:03:37.820009 8232 raft_consensus.cc:2454] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a621eebd4414b4da0d5dac00f35bee0 in term 0.
I20260924 04:03:37.820406 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a5b29c72414818a6b56a84b7ac6f86" candidate_uuid: "1a621eebd4414b4da0d5dac00f35bee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" is_pre_election: true
I20260924 04:03:37.820523 8303 leader_election.cc:302] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a621eebd4414b4da0d5dac00f35bee0, 2d09b566136849aaa498bb26586f4fa9; no voters:
I20260924 04:03:37.820698 8092 raft_consensus.cc:2454] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a621eebd4414b4da0d5dac00f35bee0 in term 0.
I20260924 04:03:37.820808 8452 raft_consensus.cc:2781] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:37.820904 8452 raft_consensus.cc:482] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:37.820956 8452 raft_consensus.cc:3037] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:37.822023 8452 raft_consensus.cc:504] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.822340 8452 leader_election.cc:288] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [CANDIDATE]: Term 1 election: Requested vote from peers e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811), 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641)
I20260924 04:03:37.822686 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a5b29c72414818a6b56a84b7ac6f86" candidate_uuid: "1a621eebd4414b4da0d5dac00f35bee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c"
I20260924 04:03:37.822742 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a5b29c72414818a6b56a84b7ac6f86" candidate_uuid: "1a621eebd4414b4da0d5dac00f35bee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d09b566136849aaa498bb26586f4fa9"
I20260924 04:03:37.822790 8092 raft_consensus.cc:3037] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:37.822862 8232 raft_consensus.cc:3037] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:37.823786 8092 raft_consensus.cc:2454] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a621eebd4414b4da0d5dac00f35bee0 in term 1.
I20260924 04:03:37.824092 8305 leader_election.cc:302] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a621eebd4414b4da0d5dac00f35bee0, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:37.824257 8232 raft_consensus.cc:2454] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a621eebd4414b4da0d5dac00f35bee0 in term 1.
I20260924 04:03:37.824270 8452 raft_consensus.cc:2781] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:37.824496 8452 raft_consensus.cc:686] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 LEADER]: Becoming Leader. State: Replica: 1a621eebd4414b4da0d5dac00f35bee0, State: Running, Role: LEADER
I20260924 04:03:37.824849 8452 consensus_queue.cc:237] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.827792 7933 catalog_manager.cc:5495] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1a621eebd4414b4da0d5dac00f35bee0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } health_report { overall_health: HEALTHY } } }
I20260924 04:03:37.860198 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:37.856736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.0.98.3:40113)
0924 04:03:37.856847 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:37.856850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:37.856863 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:37.856932 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:37.856934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:37.856991 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:37.857089 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:37.857340 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.857888 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.858775 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:37.859005 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:37.859094 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:37.859340 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:37.859811 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:37.859942 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:37.859982 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 04:03:37.860041 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 04:03:37.861833 8419 tablet.cc:2369] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:37.866084 8092 raft_consensus.cc:1261] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Refusing update from remote peer 1a621eebd4414b4da0d5dac00f35bee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:37.866083 8232 raft_consensus.cc:1261] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Refusing update from remote peer 1a621eebd4414b4da0d5dac00f35bee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:37.866727 8452 consensus_queue.cc:1035] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:37.866916 8456 consensus_queue.cc:1035] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:37.874586 8460 mvcc.cc:205] Tried to move back new op lower bound from 7332751842772410368 to 7332751842613334016. Current Snapshot: MvccSnapshot[applied={T|T < 7332751842772410368}]
I20260924 04:03:37.882366 7933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:37.882576 7933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:37.885115 7933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: Sending DeleteTablet for 3 replicas of tablet 40a5b29c72414818a6b56a84b7ac6f86
I20260924 04:03:37.885991 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a5b29c72414818a6b56a84b7ac6f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46222
I20260924 04:03:37.886101 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a5b29c72414818a6b56a84b7ac6f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55432
I20260924 04:03:37.886722 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a5b29c72414818a6b56a84b7ac6f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56180
I20260924 04:03:37.887192 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:37.887575 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:37.888114 8469 tablet_replica.cc:330] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: stopping tablet replica
I20260924 04:03:37.888445 8469 raft_consensus.cc:2229] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:37.888772 8469 raft_consensus.cc:2258] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:37.889096 8470 tablet_replica.cc:330] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: stopping tablet replica
I20260924 04:03:37.889384 8470 raft_consensus.cc:2229] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:37.889653 8470 raft_consensus.cc:2258] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:37.890507 8471 tablet_replica.cc:330] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: stopping tablet replica
I20260924 04:03:37.890816 8471 raft_consensus.cc:2229] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:37.891078 8470 ts_tablet_manager.cc:1905] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:37.891319 8471 raft_consensus.cc:2258] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:37.893150 8072 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4b595e23643978aea0408e6a48f03 (DEFAULT_TABLE table=test-table [id=110e553fdb4743deba7cfb8b0c7bc7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:37.893411 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df4b595e23643978aea0408e6a48f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:37.893836 8470 ts_tablet_manager.cc:1918] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:37.893918 8470 log.cc:1196] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/40a5b29c72414818a6b56a84b7ac6f86
I20260924 04:03:37.894192 8470 ts_tablet_manager.cc:1939] T 40a5b29c72414818a6b56a84b7ac6f86 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting consensus metadata
I20260924 04:03:37.895602 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4b595e23643978aea0408e6a48f03 (DEFAULT_TABLE table=test-table [id=110e553fdb4743deba7cfb8b0c7bc7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:37.895825 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df4b595e23643978aea0408e6a48f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:37.896035 8447 tablet_bootstrap.cc:492] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: Bootstrap starting.
I20260924 04:03:37.897387 8469 ts_tablet_manager.cc:1905] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:37.897599 7921 catalog_manager.cc:4841] TS e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811): tablet 40a5b29c72414818a6b56a84b7ac6f86 (table test-table [id=c76401badd794ede837a817385b53b2a]) successfully deleted
I20260924 04:03:37.898269 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4b595e23643978aea0408e6a48f03 (DEFAULT_TABLE table=test-table [id=110e553fdb4743deba7cfb8b0c7bc7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:37.898504 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df4b595e23643978aea0408e6a48f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:37.900010 8469 ts_tablet_manager.cc:1918] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:37.900094 8469 log.cc:1196] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/40a5b29c72414818a6b56a84b7ac6f86
I20260924 04:03:37.900452 8469 ts_tablet_manager.cc:1939] T 40a5b29c72414818a6b56a84b7ac6f86 P 2d09b566136849aaa498bb26586f4fa9: Deleting consensus metadata
I20260924 04:03:37.900698 8448 tablet_bootstrap.cc:492] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: Bootstrap starting.
I20260924 04:03:37.901752 8448 tablet_bootstrap.cc:654] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:37.902863 8448 tablet_bootstrap.cc:492] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: No bootstrap required, opened a new log
I20260924 04:03:37.902833 7919 catalog_manager.cc:4841] TS 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641): tablet 40a5b29c72414818a6b56a84b7ac6f86 (table test-table [id=c76401badd794ede837a817385b53b2a]) successfully deleted
I20260924 04:03:37.902946 8448 ts_tablet_manager.cc:1397] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:37.903357 8448 raft_consensus.cc:348] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.903474 8448 raft_consensus.cc:374] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:37.903504 8448 raft_consensus.cc:729] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d09b566136849aaa498bb26586f4fa9, State: Initialized, Role: FOLLOWER
I20260924 04:03:37.903618 8448 consensus_queue.cc:260] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.903870 8448 ts_tablet_manager.cc:1428] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:37.905167 8446 tablet_bootstrap.cc:492] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: Bootstrap starting.
I20260924 04:03:37.906134 8446 tablet_bootstrap.cc:654] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:37.907377 8446 tablet_bootstrap.cc:492] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: No bootstrap required, opened a new log
I20260924 04:03:37.907434 8447 tablet_bootstrap.cc:654] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:37.907460 8446 ts_tablet_manager.cc:1397] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:37.907895 8446 raft_consensus.cc:348] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.908051 8446 raft_consensus.cc:374] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:37.908089 8446 raft_consensus.cc:729] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a621eebd4414b4da0d5dac00f35bee0, State: Initialized, Role: FOLLOWER
I20260924 04:03:37.908206 8446 consensus_queue.cc:260] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.908465 8446 ts_tablet_manager.cc:1428] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:37.908795 8447 tablet_bootstrap.cc:492] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: No bootstrap required, opened a new log
I20260924 04:03:37.908887 8447 ts_tablet_manager.cc:1397] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260924 04:03:37.909320 8447 raft_consensus.cc:348] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.909925 8447 raft_consensus.cc:374] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:37.909968 8447 raft_consensus.cc:729] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a7481f0e914a44a1c2db3a0aeffc6c, State: Initialized, Role: FOLLOWER
I20260924 04:03:37.910087 8447 consensus_queue.cc:260] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.910853 8447 ts_tablet_manager.cc:1428] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:37.911796 8471 ts_tablet_manager.cc:1905] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:37.914402 8471 ts_tablet_manager.cc:1918] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:37.914479 8471 log.cc:1196] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/40a5b29c72414818a6b56a84b7ac6f86
I20260924 04:03:37.914677 8471 ts_tablet_manager.cc:1939] T 40a5b29c72414818a6b56a84b7ac6f86 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting consensus metadata
I20260924 04:03:37.915719 7920 catalog_manager.cc:4841] TS 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113): tablet 40a5b29c72414818a6b56a84b7ac6f86 (table test-table [id=c76401badd794ede837a817385b53b2a]) successfully deleted
W20260924 04:03:37.924456 8139 tablet.cc:2369] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:37.975741 8464 raft_consensus.cc:482] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:37.975952 8464 raft_consensus.cc:504] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.976446 8464 leader_election.cc:288] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811), 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641)
I20260924 04:03:37.976886 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b595e23643978aea0408e6a48f03" candidate_uuid: "1a621eebd4414b4da0d5dac00f35bee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" is_pre_election: true
I20260924 04:03:37.976902 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b595e23643978aea0408e6a48f03" candidate_uuid: "1a621eebd4414b4da0d5dac00f35bee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d09b566136849aaa498bb26586f4fa9" is_pre_election: true
I20260924 04:03:37.977028 8092 raft_consensus.cc:2454] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a621eebd4414b4da0d5dac00f35bee0 in term 0.
I20260924 04:03:37.977042 8232 raft_consensus.cc:2454] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a621eebd4414b4da0d5dac00f35bee0 in term 0.
I20260924 04:03:37.977360 8305 leader_election.cc:302] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a621eebd4414b4da0d5dac00f35bee0, 2d09b566136849aaa498bb26586f4fa9, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:37.977525 8464 raft_consensus.cc:2781] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:37.977583 8464 raft_consensus.cc:482] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:37.977622 8464 raft_consensus.cc:3037] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:37.978667 8464 raft_consensus.cc:504] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.979004 8464 leader_election.cc:288] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [CANDIDATE]: Term 1 election: Requested vote from peers e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811), 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641)
I20260924 04:03:37.979360 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b595e23643978aea0408e6a48f03" candidate_uuid: "1a621eebd4414b4da0d5dac00f35bee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c"
I20260924 04:03:37.979466 8092 raft_consensus.cc:3037] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:37.980422 8092 raft_consensus.cc:2454] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a621eebd4414b4da0d5dac00f35bee0 in term 1.
I20260924 04:03:37.980695 8305 leader_election.cc:302] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a621eebd4414b4da0d5dac00f35bee0, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:37.980865 8464 raft_consensus.cc:2781] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:37.980834 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b595e23643978aea0408e6a48f03" candidate_uuid: "1a621eebd4414b4da0d5dac00f35bee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d09b566136849aaa498bb26586f4fa9"
I20260924 04:03:37.980931 8464 raft_consensus.cc:686] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 LEADER]: Becoming Leader. State: Replica: 1a621eebd4414b4da0d5dac00f35bee0, State: Running, Role: LEADER
I20260924 04:03:37.980973 8232 raft_consensus.cc:3037] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:37.981065 8464 consensus_queue.cc:237] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:37.982110 8232 raft_consensus.cc:2454] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a621eebd4414b4da0d5dac00f35bee0 in term 1.
I20260924 04:03:37.982801 7935 catalog_manager.cc:5495] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1a621eebd4414b4da0d5dac00f35bee0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } health_report { overall_health: HEALTHY } } }
Sep 24 04:03:38 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:38.031129 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.022045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.0.98.62:43883)
0924 04:03:38.022154 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.022158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.022397 (+ 239us) server_negotiation.cc:407] Connection header received
0924 04:03:38.022519 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.022521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.022561 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.022627 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:38.023475 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.024052 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.025196 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.025408 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.025491 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.028076 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:38.028079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:38.028081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:38.028102 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:38.029464 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:38.029976 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:38.029978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:38.029980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:38.030045 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:38.030282 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:38.030284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:38.030286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:38.030439 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:38.030508 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.030818 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.030984 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":37}
I20260924 04:03:38.031138 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.021962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43883 (local address 127.0.0.1:34298)
0924 04:03:38.022303 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.022319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.022441 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.022678 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.022681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.023236 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:38.023359 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.023367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.024148 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.024152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.025063 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.025073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.025516 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.025519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.025637 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.026200 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:38.026214 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:38.027958 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:38.029565 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:38.029575 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:38.029581 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:38.029887 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:38.030115 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:38.030118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:38.030120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:38.030207 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:38.030519 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:38.030525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:38.030651 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.030776 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.030930 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260924 04:03:38.034862 7935 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34298:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:03:38.035554 7935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:38.040261 8072 tablet_service.cc:1467] Processing CreateTablet for tablet 01eb104f7d004d6f89a8fb3396bae099 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e61aadb5c664f08b671b313ff0193c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:38.040318 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 01eb104f7d004d6f89a8fb3396bae099 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e61aadb5c664f08b671b313ff0193c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:38.040326 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 01eb104f7d004d6f89a8fb3396bae099 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e61aadb5c664f08b671b313ff0193c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:03:38.040632 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01eb104f7d004d6f89a8fb3396bae099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.040680 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01eb104f7d004d6f89a8fb3396bae099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.040680 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01eb104f7d004d6f89a8fb3396bae099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.043002 8447 tablet_bootstrap.cc:492] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: Bootstrap starting.
I20260924 04:03:38.043253 8446 tablet_bootstrap.cc:492] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: Bootstrap starting.
I20260924 04:03:38.043895 8448 tablet_bootstrap.cc:492] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: Bootstrap starting.
I20260924 04:03:38.044039 8446 tablet_bootstrap.cc:654] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.044039 8447 tablet_bootstrap.cc:654] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.044775 8448 tablet_bootstrap.cc:654] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.045176 8447 tablet_bootstrap.cc:492] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: No bootstrap required, opened a new log
I20260924 04:03:38.045270 8447 ts_tablet_manager.cc:1397] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 04:03:38.045740 8447 raft_consensus.cc:348] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.045866 8447 raft_consensus.cc:374] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.045876 8448 tablet_bootstrap.cc:492] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: No bootstrap required, opened a new log
I20260924 04:03:38.045902 8447 raft_consensus.cc:729] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a7481f0e914a44a1c2db3a0aeffc6c, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.045925 8446 tablet_bootstrap.cc:492] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: No bootstrap required, opened a new log
I20260924 04:03:38.045944 8448 ts_tablet_manager.cc:1397] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:03:38.045997 8446 ts_tablet_manager.cc:1397] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 04:03:38.046025 8447 consensus_queue.cc:260] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.046326 8447 ts_tablet_manager.cc:1428] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:03:38.046360 8448 raft_consensus.cc:348] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.046360 8446 raft_consensus.cc:348] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.046464 8448 raft_consensus.cc:374] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.046468 8446 raft_consensus.cc:374] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.046501 8446 raft_consensus.cc:729] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a621eebd4414b4da0d5dac00f35bee0, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.046501 8448 raft_consensus.cc:729] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d09b566136849aaa498bb26586f4fa9, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.046615 8448 consensus_queue.cc:260] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.046612 8446 consensus_queue.cc:260] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.046861 8448 ts_tablet_manager.cc:1428] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:03:38.046994 8446 ts_tablet_manager.cc:1428] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:03:38.046968 8456 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.047091 8456 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:38.047168 8454 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.047184 8453 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.047326 8453 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:38.047324 8454 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:38.072165 8454 raft_consensus.cc:482] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:38.072314 8454 raft_consensus.cc:504] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.073372 8454 leader_election.cc:288] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811), 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113)
I20260924 04:03:38.077127 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.073678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55779 (local address 127.0.98.1:39811)
0924 04:03:38.073843 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.073847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.073939 (+ 92us) server_negotiation.cc:407] Connection header received
0924 04:03:38.074084 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.074087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.074128 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.074202 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:38.074529 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.075124 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.076253 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.076632 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.076724 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.076787 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.076947 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.077010 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:03:38.077152 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.073568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39811 (local address 127.0.98.2:55779)
0924 04:03:38.073854 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.073868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.073963 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.074235 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.074238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.074261 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:38.074388 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.074395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.075221 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.075224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.076143 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.076152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.076740 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.076743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.076843 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.076861 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.076924 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.076993 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":16,"thread_start_us":93,"threads_started":1}
I20260924 04:03:38.077291 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.073672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38343 (local address 127.0.98.3:40113)
0924 04:03:38.073812 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.073817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.074334 (+ 517us) server_negotiation.cc:407] Connection header received
0924 04:03:38.074460 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.074464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.074507 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.074591 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:38.074846 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.075344 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.076399 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.076797 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.076887 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.076944 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.077104 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.077174 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:03:38.077385 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.073568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40113 (local address 127.0.98.2:38343)
0924 04:03:38.074222 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.074235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.074370 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.074596 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.074599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.074618 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:38.074726 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.074731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.075446 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.075449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.076290 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.076299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.076899 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.076902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.076996 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.077012 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.077072 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.077136 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":306,"thread_start_us":85,"threads_started":1}
I20260924 04:03:38.077670 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01eb104f7d004d6f89a8fb3396bae099" candidate_uuid: "2d09b566136849aaa498bb26586f4fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" is_pre_election: true
I20260924 04:03:38.077831 8092 raft_consensus.cc:2454] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d09b566136849aaa498bb26586f4fa9 in term 0.
I20260924 04:03:38.077816 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01eb104f7d004d6f89a8fb3396bae099" candidate_uuid: "2d09b566136849aaa498bb26586f4fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a621eebd4414b4da0d5dac00f35bee0" is_pre_election: true
I20260924 04:03:38.078037 8372 raft_consensus.cc:2454] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d09b566136849aaa498bb26586f4fa9 in term 0.
I20260924 04:03:38.078181 8165 leader_election.cc:302] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d09b566136849aaa498bb26586f4fa9, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:38.078438 8454 raft_consensus.cc:2781] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:38.078518 8454 raft_consensus.cc:482] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:38.078586 8454 raft_consensus.cc:3037] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.079593 8454 raft_consensus.cc:504] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.079962 8454 leader_election.cc:288] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [CANDIDATE]: Term 1 election: Requested vote from peers e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811), 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113)
I20260924 04:03:38.080273 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01eb104f7d004d6f89a8fb3396bae099" candidate_uuid: "2d09b566136849aaa498bb26586f4fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a621eebd4414b4da0d5dac00f35bee0"
I20260924 04:03:38.080391 8372 raft_consensus.cc:3037] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.080420 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01eb104f7d004d6f89a8fb3396bae099" candidate_uuid: "2d09b566136849aaa498bb26586f4fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c"
I20260924 04:03:38.080533 8092 raft_consensus.cc:3037] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.081532 8372 raft_consensus.cc:2454] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d09b566136849aaa498bb26586f4fa9 in term 1.
I20260924 04:03:38.081532 8092 raft_consensus.cc:2454] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d09b566136849aaa498bb26586f4fa9 in term 1.
I20260924 04:03:38.081815 8165 leader_election.cc:302] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d09b566136849aaa498bb26586f4fa9, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:38.081983 8454 raft_consensus.cc:2781] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:38.082196 8454 raft_consensus.cc:686] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [term 1 LEADER]: Becoming Leader. State: Replica: 2d09b566136849aaa498bb26586f4fa9, State: Running, Role: LEADER
I20260924 04:03:38.082434 8454 consensus_queue.cc:237] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.083652 8487 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: TxnStatusTablet state changed. Reason: New leader 2d09b566136849aaa498bb26586f4fa9. Latest consensus state: current_term: 1 leader_uuid: "2d09b566136849aaa498bb26586f4fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.083755 8487 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:38.084215 8489 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:03:38.084369 8489 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:03:38.084753 7933 catalog_manager.cc:5495] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2d09b566136849aaa498bb26586f4fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20260924 04:03:38.102618 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.099105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.0.98.2:33641)
0924 04:03:38.099235 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.099239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.099254 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:03:38.099448 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.099450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.099503 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.099582 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:38.099835 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.100375 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.101201 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.101420 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.101505 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.101783 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.102225 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.102333 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.102417 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.102473 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:03:38.102618 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.098960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33641 (local address 127.0.0.1:55436)
0924 04:03:38.099193 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.099208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.099314 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.099599 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.099602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.099613 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:38.099733 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.099739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.100475 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.100479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.101093 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.101101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.101512 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.101515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.101607 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.101654 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.102325 (+ 671us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.102340 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.102396 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.102487 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260924 04:03:38.105716 8092 raft_consensus.cc:1261] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Refusing update from remote peer 2d09b566136849aaa498bb26586f4fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:38.105975 8372 raft_consensus.cc:1261] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Refusing update from remote peer 2d09b566136849aaa498bb26586f4fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:38.106518 8454 consensus_queue.cc:1035] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:38.106784 8487 consensus_queue.cc:1035] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:38.108580 8453 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: TxnStatusTablet state changed. Reason: New leader 2d09b566136849aaa498bb26586f4fa9. Latest consensus state: current_term: 1 leader_uuid: "2d09b566136849aaa498bb26586f4fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.108613 8456 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: TxnStatusTablet state changed. Reason: New leader 2d09b566136849aaa498bb26586f4fa9. Latest consensus state: current_term: 1 leader_uuid: "2d09b566136849aaa498bb26586f4fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.108704 8453 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:38.108738 8456 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:38.111649 8453 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d09b566136849aaa498bb26586f4fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.111773 8453 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P e2a7481f0e914a44a1c2db3a0aeffc6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:38.111928 8487 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d09b566136849aaa498bb26586f4fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.112056 8487 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:38.112190 8487 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d09b566136849aaa498bb26586f4fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.112283 8487 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P 2d09b566136849aaa498bb26586f4fa9: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:03:38.112865 8456 tablet_replica.cc:439] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d09b566136849aaa498bb26586f4fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } } }
I20260924 04:03:38.112998 8456 tablet_replica.cc:442] T 01eb104f7d004d6f89a8fb3396bae099 P 1a621eebd4414b4da0d5dac00f35bee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:03:38.116441 8459 mvcc.cc:205] Tried to move back new op lower bound from 7332751843755585536 to 7332751843668324352. Current Snapshot: MvccSnapshot[applied={T|T < 7332751843755585536 or (T in {7332751843755585536})}]
I20260924 04:03:38.127767 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.124227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33641 (local address 127.0.98.3:59711)
0924 04:03:38.124466 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.124479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.124567 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.124832 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.124833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.124840 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:38.124926 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.124930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.125886 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.125888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.126490 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.126497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.127054 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.127056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.127121 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.127164 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.127494 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.127501 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.127567 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.127646 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260924 04:03:38.127821 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.124334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59711 (local address 127.0.98.2:33641)
0924 04:03:38.124458 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.124463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.124546 (+ 83us) server_negotiation.cc:407] Connection header received
0924 04:03:38.124656 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.124660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.124734 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.124818 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:38.125006 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.125781 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.126590 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.126948 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.127066 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.127279 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.127366 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.127442 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.127620 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.127674 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:03:38.132376 8279 tablet.cc:2369] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:03:38.134770 8092 raft_consensus.cc:1261] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Refusing update from remote peer 1a621eebd4414b4da0d5dac00f35bee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:38.135174 8232 raft_consensus.cc:1261] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Refusing update from remote peer 1a621eebd4414b4da0d5dac00f35bee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:38.135505 8456 consensus_queue.cc:1035] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:38.135803 8464 consensus_queue.cc:1035] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:38.145195 8461 mvcc.cc:205] Tried to move back new op lower bound from 7332751843876753408 to 7332751843251773440. Current Snapshot: MvccSnapshot[applied={T|T < 7332751843876753408}]
I20260924 04:03:38.158179 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.154339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40113 (local address 127.0.98.2:52911)
0924 04:03:38.154644 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.154657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.154742 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.154949 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.154950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.154956 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:38.155094 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.155100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.156102 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.156105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.156870 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.156879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.157460 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.157463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.157535 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.157583 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.157899 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.157903 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.157948 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.158018 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":115,"threads_started":1}
I20260924 04:03:38.158182 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.154412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52911 (local address 127.0.98.3:40113)
0924 04:03:38.154523 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.154528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.154714 (+ 186us) server_negotiation.cc:407] Connection header received
0924 04:03:38.154812 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.154815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.154865 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.154942 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:38.155190 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.156002 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.156996 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.157367 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.157465 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.157704 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.157825 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.157892 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.157961 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.158019 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:03:38.177990 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:38.178164 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:38.180275 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: Sending DeleteTablet for 3 replicas of tablet 3df4b595e23643978aea0408e6a48f03
I20260924 04:03:38.180964 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df4b595e23643978aea0408e6a48f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56180
I20260924 04:03:38.180986 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df4b595e23643978aea0408e6a48f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46222
I20260924 04:03:38.181144 8470 tablet_replica.cc:330] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: stopping tablet replica
I20260924 04:03:38.181147 8471 tablet_replica.cc:330] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: stopping tablet replica
I20260924 04:03:38.181244 8471 raft_consensus.cc:2229] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:38.181239 8470 raft_consensus.cc:2229] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:38.181344 8470 raft_consensus.cc:2258] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.181413 8471 raft_consensus.cc:2258] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.181685 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df4b595e23643978aea0408e6a48f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55432
I20260924 04:03:38.181843 8469 tablet_replica.cc:330] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: stopping tablet replica
I20260924 04:03:38.181931 8469 raft_consensus.cc:2229] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:38.182016 8469 raft_consensus.cc:2258] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.182165 8471 ts_tablet_manager.cc:1905] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.182940 8469 ts_tablet_manager.cc:1905] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.185061 8471 ts_tablet_manager.cc:1918] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:38.185137 8471 log.cc:1196] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/3df4b595e23643978aea0408e6a48f03
I20260924 04:03:38.185365 8471 ts_tablet_manager.cc:1939] T 3df4b595e23643978aea0408e6a48f03 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting consensus metadata
I20260924 04:03:38.185487 8469 ts_tablet_manager.cc:1918] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:38.185547 8469 log.cc:1196] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/3df4b595e23643978aea0408e6a48f03
I20260924 04:03:38.185757 8469 ts_tablet_manager.cc:1939] T 3df4b595e23643978aea0408e6a48f03 P 2d09b566136849aaa498bb26586f4fa9: Deleting consensus metadata
I20260924 04:03:38.186362 7920 catalog_manager.cc:4841] TS 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113): tablet 3df4b595e23643978aea0408e6a48f03 (table test-table [id=110e553fdb4743deba7cfb8b0c7bc7ed]) successfully deleted
I20260924 04:03:38.187052 7919 catalog_manager.cc:4841] TS 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641): tablet 3df4b595e23643978aea0408e6a48f03 (table test-table [id=110e553fdb4743deba7cfb8b0c7bc7ed]) successfully deleted
I20260924 04:03:38.190433 8470 ts_tablet_manager.cc:1905] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.193076 8470 ts_tablet_manager.cc:1918] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:38.193156 8470 log.cc:1196] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/3df4b595e23643978aea0408e6a48f03
I20260924 04:03:38.193387 8470 ts_tablet_manager.cc:1939] T 3df4b595e23643978aea0408e6a48f03 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting consensus metadata
I20260924 04:03:38.194027 7921 catalog_manager.cc:4841] TS e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811): tablet 3df4b595e23643978aea0408e6a48f03 (table test-table [id=110e553fdb4743deba7cfb8b0c7bc7ed]) successfully deleted
Sep 24 04:03:38 dist-test-slave-zcsk krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:03:38 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:38 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:03:38.217042 7984 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:38.210830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.98.62:43883)
0924 04:03:38.210951 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.210956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.211149 (+ 193us) server_negotiation.cc:407] Connection header received
0924 04:03:38.211288 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.211291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.211337 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.211405 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:38.212138 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.212661 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.213496 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.213711 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.213797 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.216917 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34314: BlockingRecv error: recv got EOF from 127.0.0.1:34314 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 04:03:38 dist-test-slave-zcsk krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:03:38.226668 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.218670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.0.98.62:43883)
0924 04:03:38.218825 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.218830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.218985 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:03:38.219090 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.219094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.219137 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.219202 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:38.219743 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.220307 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.221071 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.221294 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.221377 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.223808 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:38.223812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:38.223814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:38.223840 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:38.225147 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:38.225516 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:38.225519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:38.225522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:38.225594 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:38.225831 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:38.225834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:38.225836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:38.226071 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:38.226186 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.226413 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.226520 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:03:38.229287 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:38.229701 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:38.234337 8072 tablet_service.cc:1467] Processing CreateTablet for tablet d72ac81c069d41749766c91e74204788 (DEFAULT_TABLE table=test-table [id=ee5a5af6b7f54a9d8fa88c219fe018be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:38.234362 8212 tablet_service.cc:1467] Processing CreateTablet for tablet d72ac81c069d41749766c91e74204788 (DEFAULT_TABLE table=test-table [id=ee5a5af6b7f54a9d8fa88c219fe018be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:38.234640 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72ac81c069d41749766c91e74204788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.234639 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72ac81c069d41749766c91e74204788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.234613 8352 tablet_service.cc:1467] Processing CreateTablet for tablet d72ac81c069d41749766c91e74204788 (DEFAULT_TABLE table=test-table [id=ee5a5af6b7f54a9d8fa88c219fe018be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:38.234889 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72ac81c069d41749766c91e74204788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.237176 8448 tablet_bootstrap.cc:492] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: Bootstrap starting.
I20260924 04:03:38.237324 8447 tablet_bootstrap.cc:492] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: Bootstrap starting.
I20260924 04:03:38.237973 8448 tablet_bootstrap.cc:654] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.238273 8447 tablet_bootstrap.cc:654] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.238889 8446 tablet_bootstrap.cc:492] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: Bootstrap starting.
I20260924 04:03:38.239343 8448 tablet_bootstrap.cc:492] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: No bootstrap required, opened a new log
I20260924 04:03:38.239424 8448 ts_tablet_manager.cc:1397] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:03:38.239673 8446 tablet_bootstrap.cc:654] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.240263 8448 raft_consensus.cc:348] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.240381 8448 raft_consensus.cc:374] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.240407 8448 raft_consensus.cc:729] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d09b566136849aaa498bb26586f4fa9, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.240520 8448 consensus_queue.cc:260] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.240674 8446 tablet_bootstrap.cc:492] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: No bootstrap required, opened a new log
I20260924 04:03:38.240753 8446 ts_tablet_manager.cc:1397] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:38.241113 8446 raft_consensus.cc:348] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.241218 8446 raft_consensus.cc:374] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.241245 8446 raft_consensus.cc:729] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a621eebd4414b4da0d5dac00f35bee0, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.241338 8448 ts_tablet_manager.cc:1428] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:38.241353 8446 consensus_queue.cc:260] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.241572 8446 ts_tablet_manager.cc:1428] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:38.242063 8447 tablet_bootstrap.cc:492] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: No bootstrap required, opened a new log
I20260924 04:03:38.242166 8447 ts_tablet_manager.cc:1397] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260924 04:03:38.242568 8447 raft_consensus.cc:348] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.242672 8447 raft_consensus.cc:374] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.242698 8447 raft_consensus.cc:729] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a7481f0e914a44a1c2db3a0aeffc6c, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.242808 8447 consensus_queue.cc:260] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.243034 8447 ts_tablet_manager.cc:1428] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:03:38.376039 8453 raft_consensus.cc:482] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:38.376185 8453 raft_consensus.cc:504] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.377221 8453 leader_election.cc:288] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641), 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113)
I20260924 04:03:38.381798 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.377595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59045 (local address 127.0.98.2:33641)
0924 04:03:38.377728 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.377733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.378546 (+ 813us) server_negotiation.cc:407] Connection header received
0924 04:03:38.378641 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.378644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.378687 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.378745 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:38.379006 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.379568 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.380627 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.381228 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.381336 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.381405 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.381604 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.381676 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:03:38.381829 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.377533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33641 (local address 127.0.98.1:59045)
0924 04:03:38.378456 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.378470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.378560 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.378779 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.378782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.378797 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:38.378907 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.378912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.379676 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.379679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.380514 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.380523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.381377 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.381381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.381477 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.381502 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.381581 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.381653 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":3,"thread_start_us":630,"threads_started":1}
I20260924 04:03:38.381857 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.377617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50313 (local address 127.0.98.3:40113)
0924 04:03:38.377856 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.377861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.377985 (+ 124us) server_negotiation.cc:407] Connection header received
0924 04:03:38.378169 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.378174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.378237 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.378329 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:03:38.378590 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.379209 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.380706 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.381272 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.381353 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.381404 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.381681 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.381758 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260924 04:03:38.382197 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.377526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40113 (local address 127.0.98.1:50313)
0924 04:03:38.377886 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.377902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.378017 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.378334 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.378336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.378351 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:03:38.378489 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.378495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.379365 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.379368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.380598 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.380609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.381464 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.381467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.381539 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.381557 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.381975 (+ 418us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.382053 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":84,"threads_started":1}
I20260924 04:03:38.382388 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ac81c069d41749766c91e74204788" candidate_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d09b566136849aaa498bb26586f4fa9" is_pre_election: true
I20260924 04:03:38.382541 8232 raft_consensus.cc:2454] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a7481f0e914a44a1c2db3a0aeffc6c in term 0.
I20260924 04:03:38.382586 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ac81c069d41749766c91e74204788" candidate_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a621eebd4414b4da0d5dac00f35bee0" is_pre_election: true
I20260924 04:03:38.382709 8372 raft_consensus.cc:2454] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a7481f0e914a44a1c2db3a0aeffc6c in term 0.
I20260924 04:03:38.382887 8023 leader_election.cc:302] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d09b566136849aaa498bb26586f4fa9, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:38.383181 8453 raft_consensus.cc:2781] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:38.383253 8453 raft_consensus.cc:482] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:38.383294 8453 raft_consensus.cc:3037] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.384485 8453 raft_consensus.cc:504] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.384840 8453 leader_election.cc:288] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [CANDIDATE]: Term 1 election: Requested vote from peers 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641), 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113)
I20260924 04:03:38.385159 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ac81c069d41749766c91e74204788" candidate_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d09b566136849aaa498bb26586f4fa9"
I20260924 04:03:38.385281 8232 raft_consensus.cc:3037] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.385272 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ac81c069d41749766c91e74204788" candidate_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a621eebd4414b4da0d5dac00f35bee0"
I20260924 04:03:38.385380 8372 raft_consensus.cc:3037] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.386171 8232 raft_consensus.cc:2454] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a7481f0e914a44a1c2db3a0aeffc6c in term 1.
I20260924 04:03:38.386227 8372 raft_consensus.cc:2454] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a7481f0e914a44a1c2db3a0aeffc6c in term 1.
I20260924 04:03:38.386489 8024 leader_election.cc:302] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a621eebd4414b4da0d5dac00f35bee0, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:38.386672 8453 raft_consensus.cc:2781] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:38.386909 8453 raft_consensus.cc:686] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 LEADER]: Becoming Leader. State: Replica: e2a7481f0e914a44a1c2db3a0aeffc6c, State: Running, Role: LEADER
I20260924 04:03:38.387174 8453 consensus_queue.cc:237] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.389417 7933 catalog_manager.cc:5495] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c reported cstate change: term changed from 0 to 1, leader changed from <none> to e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20260924 04:03:38.435027 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.431591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.0.98.1:39811)
0924 04:03:38.431712 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.431717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.431731 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:38.431863 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.431866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.431953 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.432050 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:38.432287 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.432781 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.433627 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.433844 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.433931 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.434193 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.434651 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.434754 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.434834 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.434890 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:38.437518 8232 raft_consensus.cc:1261] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Refusing update from remote peer e2a7481f0e914a44a1c2db3a0aeffc6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:38.437546 8372 raft_consensus.cc:1261] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Refusing update from remote peer e2a7481f0e914a44a1c2db3a0aeffc6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:38.437994 8520 consensus_queue.cc:1035] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:38.438167 8453 consensus_queue.cc:1035] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:38.449429 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:38.449579 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:38.451690 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: Sending DeleteTablet for 3 replicas of tablet d72ac81c069d41749766c91e74204788
I20260924 04:03:38.452333 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet d72ac81c069d41749766c91e74204788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46222
I20260924 04:03:38.452310 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet d72ac81c069d41749766c91e74204788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55432
I20260924 04:03:38.452519 8470 tablet_replica.cc:330] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: stopping tablet replica
I20260924 04:03:38.452632 8470 raft_consensus.cc:2229] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:38.452926 8470 raft_consensus.cc:2258] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.453114 8469 tablet_replica.cc:330] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: stopping tablet replica
I20260924 04:03:38.453240 8469 raft_consensus.cc:2229] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:38.453361 8469 raft_consensus.cc:2258] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.453871 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet d72ac81c069d41749766c91e74204788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56180
I20260924 04:03:38.454005 8470 ts_tablet_manager.cc:1905] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.454047 8471 tablet_replica.cc:330] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: stopping tablet replica
I20260924 04:03:38.454150 8471 raft_consensus.cc:2229] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:38.454207 8469 ts_tablet_manager.cc:1905] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.454279 8471 raft_consensus.cc:2258] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.455160 8471 ts_tablet_manager.cc:1905] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.456704 8470 ts_tablet_manager.cc:1918] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:38.456784 8470 log.cc:1196] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/d72ac81c069d41749766c91e74204788
I20260924 04:03:38.456758 8469 ts_tablet_manager.cc:1918] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:38.456831 8469 log.cc:1196] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/d72ac81c069d41749766c91e74204788
I20260924 04:03:38.457032 8470 ts_tablet_manager.cc:1939] T d72ac81c069d41749766c91e74204788 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting consensus metadata
I20260924 04:03:38.457050 8469 ts_tablet_manager.cc:1939] T d72ac81c069d41749766c91e74204788 P 2d09b566136849aaa498bb26586f4fa9: Deleting consensus metadata
I20260924 04:03:38.457635 8471 ts_tablet_manager.cc:1918] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:03:38.457702 8471 log.cc:1196] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/d72ac81c069d41749766c91e74204788
I20260924 04:03:38.457792 7921 catalog_manager.cc:4841] TS e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811): tablet d72ac81c069d41749766c91e74204788 (table test-table [id=ee5a5af6b7f54a9d8fa88c219fe018be]) successfully deleted
I20260924 04:03:38.457917 8471 ts_tablet_manager.cc:1939] T d72ac81c069d41749766c91e74204788 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting consensus metadata
I20260924 04:03:38.457803 7919 catalog_manager.cc:4841] TS 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641): tablet d72ac81c069d41749766c91e74204788 (table test-table [id=ee5a5af6b7f54a9d8fa88c219fe018be]) successfully deleted
I20260924 04:03:38.458518 7920 catalog_manager.cc:4841] TS 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113): tablet d72ac81c069d41749766c91e74204788 (table test-table [id=ee5a5af6b7f54a9d8fa88c219fe018be]) successfully deleted
I20260924 04:03:38.459558 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:03:38.460012 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:03:38.464735 8212 tablet_service.cc:1467] Processing CreateTablet for tablet d5adae388d8245a2b437275bc1bbdd01 (DEFAULT_TABLE table=test-table [id=1bd1664ee27447e1a37d9686167c618b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:38.464932 8072 tablet_service.cc:1467] Processing CreateTablet for tablet d5adae388d8245a2b437275bc1bbdd01 (DEFAULT_TABLE table=test-table [id=1bd1664ee27447e1a37d9686167c618b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:38.465073 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5adae388d8245a2b437275bc1bbdd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.465164 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5adae388d8245a2b437275bc1bbdd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.467478 8447 tablet_bootstrap.cc:492] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c: Bootstrap starting.
I20260924 04:03:38.467523 8448 tablet_bootstrap.cc:492] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: Bootstrap starting.
I20260924 04:03:38.467670 8352 tablet_service.cc:1467] Processing CreateTablet for tablet d5adae388d8245a2b437275bc1bbdd01 (DEFAULT_TABLE table=test-table [id=1bd1664ee27447e1a37d9686167c618b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:03:38.467952 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5adae388d8245a2b437275bc1bbdd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:38.468392 8448 tablet_bootstrap.cc:654] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.468506 8447 tablet_bootstrap.cc:654] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.470162 8447 tablet_bootstrap.cc:492] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c: No bootstrap required, opened a new log
I20260924 04:03:38.470259 8447 ts_tablet_manager.cc:1397] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:03:38.470528 8446 tablet_bootstrap.cc:492] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: Bootstrap starting.
I20260924 04:03:38.470745 8447 raft_consensus.cc:348] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.470866 8447 raft_consensus.cc:374] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.470896 8447 raft_consensus.cc:729] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a7481f0e914a44a1c2db3a0aeffc6c, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.471016 8447 consensus_queue.cc:260] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.471271 8447 ts_tablet_manager.cc:1428] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:03:38.471329 8446 tablet_bootstrap.cc:654] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:38.472430 8446 tablet_bootstrap.cc:492] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: No bootstrap required, opened a new log
I20260924 04:03:38.472528 8446 ts_tablet_manager.cc:1397] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:38.472932 8446 raft_consensus.cc:348] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.473049 8446 raft_consensus.cc:374] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.473084 8446 raft_consensus.cc:729] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a621eebd4414b4da0d5dac00f35bee0, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.473209 8446 consensus_queue.cc:260] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.473474 8446 ts_tablet_manager.cc:1428] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:38.475248 8448 tablet_bootstrap.cc:492] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: No bootstrap required, opened a new log
I20260924 04:03:38.475360 8448 ts_tablet_manager.cc:1397] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 04:03:38.475811 8448 raft_consensus.cc:348] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.475963 8448 raft_consensus.cc:374] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:38.476006 8448 raft_consensus.cc:729] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d09b566136849aaa498bb26586f4fa9, State: Initialized, Role: FOLLOWER
I20260924 04:03:38.476135 8448 consensus_queue.cc:260] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.476392 8448 ts_tablet_manager.cc:1428] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:03:38.536911 8525 raft_consensus.cc:482] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:03:38.537078 8525 raft_consensus.cc:504] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.537600 8525 leader_election.cc:288] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641), 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113)
I20260924 04:03:38.538015 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5adae388d8245a2b437275bc1bbdd01" candidate_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a621eebd4414b4da0d5dac00f35bee0" is_pre_election: true
I20260924 04:03:38.538036 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5adae388d8245a2b437275bc1bbdd01" candidate_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d09b566136849aaa498bb26586f4fa9" is_pre_election: true
I20260924 04:03:38.538156 8372 raft_consensus.cc:2454] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a7481f0e914a44a1c2db3a0aeffc6c in term 0.
I20260924 04:03:38.538195 8232 raft_consensus.cc:2454] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a7481f0e914a44a1c2db3a0aeffc6c in term 0.
I20260924 04:03:38.538478 8024 leader_election.cc:302] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a621eebd4414b4da0d5dac00f35bee0, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:38.538700 8525 raft_consensus.cc:2781] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:03:38.538772 8525 raft_consensus.cc:482] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:03:38.538822 8525 raft_consensus.cc:3037] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.540112 8525 raft_consensus.cc:504] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.540532 8525 leader_election.cc:288] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [CANDIDATE]: Term 1 election: Requested vote from peers 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641), 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113)
I20260924 04:03:38.540892 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5adae388d8245a2b437275bc1bbdd01" candidate_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d09b566136849aaa498bb26586f4fa9"
I20260924 04:03:38.541038 8232 raft_consensus.cc:3037] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.542197 8232 raft_consensus.cc:2454] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a7481f0e914a44a1c2db3a0aeffc6c in term 1.
I20260924 04:03:38.542476 8023 leader_election.cc:302] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d09b566136849aaa498bb26586f4fa9, e2a7481f0e914a44a1c2db3a0aeffc6c; no voters:
I20260924 04:03:38.542470 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5adae388d8245a2b437275bc1bbdd01" candidate_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a621eebd4414b4da0d5dac00f35bee0"
I20260924 04:03:38.542608 8372 raft_consensus.cc:3037] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:38.543773 8372 raft_consensus.cc:2454] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a7481f0e914a44a1c2db3a0aeffc6c in term 1.
I20260924 04:03:38.544032 8525 raft_consensus.cc:2781] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:38.544134 8525 raft_consensus.cc:686] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 LEADER]: Becoming Leader. State: Replica: e2a7481f0e914a44a1c2db3a0aeffc6c, State: Running, Role: LEADER
I20260924 04:03:38.544274 8525 consensus_queue.cc:237] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } }
I20260924 04:03:38.546106 7933 catalog_manager.cc:5495] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c reported cstate change: term changed from 0 to 1, leader changed from <none> to e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a7481f0e914a44a1c2db3a0aeffc6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20260924 04:03:38.587643 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.583685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43579 (local address 127.0.98.2:33641)
0924 04:03:38.583815 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.583818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.583986 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:03:38.584196 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.584199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.584267 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.584331 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:38.584723 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.585243 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.586528 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.586758 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.586839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.587226 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.587297 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.587343 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.587484 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.587530 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:03:38.587777 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.583531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33641 (local address 127.0.98.1:43579)
0924 04:03:38.583850 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.583868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.584051 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.584377 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.584380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.584390 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:38.584557 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.584566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.585361 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.585365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.586392 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.586405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.586890 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.586895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.587004 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.587069 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.587386 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.587393 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.587470 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.587574 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
I20260924 04:03:38.594134 8232 raft_consensus.cc:1261] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Refusing update from remote peer e2a7481f0e914a44a1c2db3a0aeffc6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:38.594152 8372 raft_consensus.cc:1261] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Refusing update from remote peer e2a7481f0e914a44a1c2db3a0aeffc6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:03:38.594513 8453 consensus_queue.cc:1035] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d09b566136849aaa498bb26586f4fa9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:38.594802 8525 consensus_queue.cc:1035] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a621eebd4414b4da0d5dac00f35bee0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:03:38.612738 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.608885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39811 (local address 127.0.98.2:52049)
0924 04:03:38.609076 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.609092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:38.609180 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:38.609562 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:38.609565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:38.609573 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:03:38.609685 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.609692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.610555 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.610558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.611375 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:38.611387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.612023 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.612027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:38.612113 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.612143 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.612419 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.612440 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:03:38.612491 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:03:38.612587 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 04:03:38.613303 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.609158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52049 (local address 127.0.98.1:39811)
0924 04:03:38.609264 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:38.609268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:38.609283 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:03:38.609328 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:38.609331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:38.609377 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:38.609448 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:03:38.609779 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.610429 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.611508 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:38.611831 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:38.611949 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:38.612253 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:03:38.612337 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:03:38.612389 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:38.613130 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 04:03:38.613183 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:03:38.633025 7933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:38.633198 7933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:03:38.635104 7933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcb596637a1049dbb101c01c7d75bf25: Sending DeleteTablet for 3 replicas of tablet d5adae388d8245a2b437275bc1bbdd01
I20260924 04:03:38.635613 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet d5adae388d8245a2b437275bc1bbdd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46222
I20260924 04:03:38.635653 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet d5adae388d8245a2b437275bc1bbdd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55432
I20260924 04:03:38.635857 8470 tablet_replica.cc:330] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c: stopping tablet replica
I20260924 04:03:38.635860 8469 tablet_replica.cc:330] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: stopping tablet replica
I20260924 04:03:38.635982 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet d5adae388d8245a2b437275bc1bbdd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:03:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56180
I20260924 04:03:38.636003 8470 raft_consensus.cc:2229] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:03:38.636003 8469 raft_consensus.cc:2229] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:38.636116 8469 raft_consensus.cc:2258] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.636132 8471 tablet_replica.cc:330] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: stopping tablet replica
I20260924 04:03:38.636185 8470 raft_consensus.cc:2258] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.636233 8471 raft_consensus.cc:2229] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:03:38.636332 8471 raft_consensus.cc:2258] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:03:38.636955 8470 ts_tablet_manager.cc:1905] T d5adae388d8245a2b437275bc1bbdd01 P e2a7481f0e914a44a1c2db3a0aeffc6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.637173 8469 ts_tablet_manager.cc:1905] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.637275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 8007
I20260924 04:03:38.639149 8471 ts_tablet_manager.cc:1905] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:03:38.639547 8469 ts_tablet_manager.cc:1918] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:38.639605 8469 log.cc:1196] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/d5adae388d8245a2b437275bc1bbdd01
I20260924 04:03:38.639811 8469 ts_tablet_manager.cc:1939] T d5adae388d8245a2b437275bc1bbdd01 P 2d09b566136849aaa498bb26586f4fa9: Deleting consensus metadata
I20260924 04:03:38.641331 8471 ts_tablet_manager.cc:1918] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:03:38.641389 8471 log.cc:1196] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/d5adae388d8245a2b437275bc1bbdd01
I20260924 04:03:38.641536 8471 ts_tablet_manager.cc:1939] T d5adae388d8245a2b437275bc1bbdd01 P 1a621eebd4414b4da0d5dac00f35bee0: Deleting consensus metadata
I20260924 04:03:38.642266 7920 catalog_manager.cc:4841] TS 1a621eebd4414b4da0d5dac00f35bee0 (127.0.98.3:40113): tablet d5adae388d8245a2b437275bc1bbdd01 (table test-table [id=1bd1664ee27447e1a37d9686167c618b]) successfully deleted
I20260924 04:03:38.644344 7919 catalog_manager.cc:4841] TS 2d09b566136849aaa498bb26586f4fa9 (127.0.98.2:33641): tablet d5adae388d8245a2b437275bc1bbdd01 (table test-table [id=1bd1664ee27447e1a37d9686167c618b]) successfully deleted
W20260924 04:03:38.646754 7921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39811 (error 108)
I20260924 04:03:38.646998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 8147
I20260924 04:03:38.650194 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:38.649812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39811 (local address 127.0.0.1:46230)
0924 04:03:38.650038 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:38.650107 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
W20260924 04:03:38.650339 7921 catalog_manager.cc:4584] TS e2a7481f0e914a44a1c2db3a0aeffc6c (127.0.98.1:39811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5adae388d8245a2b437275bc1bbdd01: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39811: connect: Connection refused (error 111)
I20260924 04:03:38.653381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 8287
I20260924 04:03:38.659435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 7903
2026-09-24T04:03:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3044 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:38 dist-test-slave-zcsk krb5kdc[8540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:38 dist-test-slave-zcsk krb5kdc[8540](info): set up 2 sockets
Sep 24 04:03:38 dist-test-slave-zcsk krb5kdc[8540](info): commencing operation
krb5kdc: starting...
W20260924 04:03:39.616571 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:39 dist-test-slave-zcsk krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:39.734822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35351
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:39.844203 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:39.844482 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:39.844524 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:39.848559 8553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:39.848626 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:39.848644 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:39.848660 8553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:39.848675 8553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:39.853976 8553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:39.854998 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:39.856010 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:39.864478 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:39.864495 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:39.864495 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:39.864552 8553 server_base.cc:1034] running on GCE node
I20260924 04:03:39.865357 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:39.866441 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:39.867614 8553 hybrid_clock.cc:648] HybridClock initialized: now 1790222619867587 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T04:03:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1213 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:39 dist-test-slave-zcsk krb5kdc[8566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:39 dist-test-slave-zcsk krb5kdc[8566](info): set up 2 sockets
Sep 24 04:03:39 dist-test-slave-zcsk krb5kdc[8566](info): commencing operation
krb5kdc: starting...
W20260924 04:03:40.832319 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:03:40 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:03:40.951306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:41.062254 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:41.062546 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:41.062584 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:41.066654 8579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:41.066720 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:41.066740 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:41.066756 8579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:41.066771 8579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:41.072180 8579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:41.073266 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:41.074272 8579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:41.082803 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:41.082948 8579 server_base.cc:1034] running on GCE node
W20260924 04:03:41.082808 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:41.082803 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:41.083707 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:41.084950 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:41.086138 8579 hybrid_clock.cc:648] HybridClock initialized: now 1790222621086103 us; error 44 us; skew 500 ppm
Sep 24 04:03:41 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:41.089012 8579 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:41.089960 8579 webserver.cc:460] Webserver started at http://127.0.98.62:45565/ using document root <none> and password file <none>
I20260924 04:03:41.090518 8579 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:41.090586 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:41.090785 8579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:41.092931 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "ad0d5347e49c466bbdc0b165cb85f15a"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "0b1e084510df819f23496b21b0c3c3bf"
server_key_iv: "5d437681ba110328bb4c669cf854c45a"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.093467 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad0d5347e49c466bbdc0b165cb85f15a"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "0b1e084510df819f23496b21b0c3c3bf"
server_key_iv: "5d437681ba110328bb4c669cf854c45a"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.097180 8579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:41.099673 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.100972 8579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:41.101107 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "ad0d5347e49c466bbdc0b165cb85f15a"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "0b1e084510df819f23496b21b0c3c3bf"
server_key_iv: "5d437681ba110328bb4c669cf854c45a"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.101207 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:41.114789 8579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:41.117806 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:41.117991 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:41.124805 8579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43579
I20260924 04:03:41.124828 8646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43579 every 8 connection(s)
I20260924 04:03:41.125770 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:41.127383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 8579
I20260924 04:03:41.127521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:41.127802 392 external_mini_cluster.cc:1428] Setting key 2134226f3af5abb50963410b9ae9e995
I20260924 04:03:41.128823 8647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:41.134325 8647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Bootstrap starting.
Sep 24 04:03:41 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:41.136787 8647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:41.137563 8647 log.cc:824] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:41.139484 8647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: No bootstrap required, opened a new log
I20260924 04:03:41.141290 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:41.129235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.0.98.62:43579)
0924 04:03:41.129719 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:41.129744 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:41.129787 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:03:41.130438 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:41.130455 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:41.130781 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:41.131100 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:41.131874 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.132893 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.133732 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.134202 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.134351 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:41.136891 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:41.136898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:41.136918 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:41.136950 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:41.138662 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:41.139065 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:41.139069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:41.139074 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:41.139176 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:41.139430 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:41.139433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:41.139435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:41.139850 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:41.140055 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:41.140303 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 04:03:41.140583 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":151,"threads_started":1}
I20260924 04:03:41.143054 8647 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260924 04:03:41.143280 8647 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:41.143380 8647 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0d5347e49c466bbdc0b165cb85f15a, State: Initialized, Role: FOLLOWER
I20260924 04:03:41.143788 8647 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260924 04:03:41.143951 8647 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:41.144017 8647 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:41.144115 8647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:41.145022 8647 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260924 04:03:41.145354 8647 leader_election.cc:302] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad0d5347e49c466bbdc0b165cb85f15a; no voters:
I20260924 04:03:41.145643 8647 leader_election.cc:288] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:41.145744 8652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:41.145965 8652 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 1 LEADER]: Becoming Leader. State: Replica: ad0d5347e49c466bbdc0b165cb85f15a, State: Running, Role: LEADER
I20260924 04:03:41.146307 8652 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260924 04:03:41.146858 8647 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:41.148666 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } } }
I20260924 04:03:41.148800 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:41.149108 8654 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad0d5347e49c466bbdc0b165cb85f15a. Latest consensus state: current_term: 1 leader_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } } }
I20260924 04:03:41.149248 8654 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:41.149439 8661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:41.152212 8661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:41.158030 8661 catalog_manager.cc:1347] Generated new cluster ID: d9cdf4d34a634886b487c9b4b8044b3d
I20260924 04:03:41.158108 8661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:41.186254 8661 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:41.187238 8661 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:41.191836 8661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Generated new TSK 0
I20260924 04:03:41.192574 8661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:03:41.220809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:41.333529 8673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:41.333855 8673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:41.333916 8673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:41.338280 8673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:41.338340 8673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:41.338425 8673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:41.343708 8673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:41.344805 8673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:41.345834 8673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:41.355609 8681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:41.355618 8679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:41.355615 8678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:41.355667 8673 server_base.cc:1034] running on GCE node
I20260924 04:03:41.356259 8673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:41.356895 8673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:41.358058 8673 hybrid_clock.cc:648] HybridClock initialized: now 1790222621358037 us; error 34 us; skew 500 ppm
Sep 24 04:03:41 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:41.360733 8673 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:03:41.361555 8673 webserver.cc:460] Webserver started at http://127.0.98.1:38089/ using document root <none> and password file <none>
I20260924 04:03:41.362074 8673 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:41.362128 8673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:41.362290 8673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:41.364329 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "8b8d17b1d2db4bd8805737321ee287f9"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "04eedca7395b0feac3b1d5f142d0a245"
server_key_iv: "e445301d9e1c1433900810a38596e79f"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.364830 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8b8d17b1d2db4bd8805737321ee287f9"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "04eedca7395b0feac3b1d5f142d0a245"
server_key_iv: "e445301d9e1c1433900810a38596e79f"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.368583 8673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:03:41.371558 8688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.372830 8673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:03:41.372952 8673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "8b8d17b1d2db4bd8805737321ee287f9"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "04eedca7395b0feac3b1d5f142d0a245"
server_key_iv: "e445301d9e1c1433900810a38596e79f"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.373066 8673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:41.389922 8673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:41.393075 8673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:41.393246 8673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:41.393836 8673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:41.394675 8673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:41.394722 8673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.394767 8673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:41.394794 8673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.403271 8673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37145
I20260924 04:03:41.403321 8801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37145 every 8 connection(s)
I20260924 04:03:41.404211 8673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:41.407245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 8673
I20260924 04:03:41.407371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:41.407675 392 external_mini_cluster.cc:1428] Setting key 2ec4f68d137125c0e99bffdb68fa886f
Sep 24 04:03:41 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:03:41.418087 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:41.405976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48953 (local address 127.0.98.62:43579)
0924 04:03:41.406110 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:41.406115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:41.407014 (+ 899us) server_negotiation.cc:407] Connection header received
0924 04:03:41.408556 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:41.408560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:41.408602 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:41.408686 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:41.410088 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.410692 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.412160 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.412438 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.412527 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:41.414830 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:41.414834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:41.414837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:41.414861 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:41.416144 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:41.416604 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:41.416606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:41.416608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:41.416677 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:41.416990 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:41.416993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:41.416994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:41.417172 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:41.417254 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:41.417798 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 04:03:41.417922 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:03:41.419041 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:41.406408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:48953)
0924 04:03:41.406846 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:41.406888 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:41.408368 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:41.408839 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:41.408851 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:41.409281 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:41.409947 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:41.409965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.410814 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.410817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:41.412033 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:41.412043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.412575 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.412579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:41.412751 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:41.413285 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:41.413301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:41.414674 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:41.416244 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:41.416250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:41.416266 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:41.416513 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:41.416755 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:41.416758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:41.416760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:41.416889 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:41.417268 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:41.417275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:41.417576 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 04:03:41.417747 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 04:03:41.418082 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260924 04:03:41.420288 8802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43579
I20260924 04:03:41.420565 8802 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:41.421128 8802 heartbeater.cc:511] Master 127.0.98.62:43579 requested a full tablet report, sending...
I20260924 04:03:41.422816 8611 ts_manager.cc:194] Registered new tserver with Master: 8b8d17b1d2db4bd8805737321ee287f9 (127.0.98.1:37145)
I20260924 04:03:41.423940 8611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:03:41.432859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:41.545307 8807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:41.545591 8807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:41.545634 8807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:41.549787 8807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:41.549856 8807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:41.549952 8807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:41.555222 8807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:41.556268 8807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:41.557220 8807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:41.566797 8813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:41.566800 8812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:41.566887 8807 server_base.cc:1034] running on GCE node
W20260924 04:03:41.566797 8815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:41.567447 8807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:41.568176 8807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:41.569342 8807 hybrid_clock.cc:648] HybridClock initialized: now 1790222621569320 us; error 36 us; skew 500 ppm
Sep 24 04:03:41 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:41.572119 8807 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:03:41.573093 8807 webserver.cc:460] Webserver started at http://127.0.98.2:38571/ using document root <none> and password file <none>
I20260924 04:03:41.573655 8807 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:41.573725 8807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:41.573938 8807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:41.576107 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "7b2c86f5cfbb4d31b27fa7697a91e7cf"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "d880110dd35a12bbab6680139cc03793"
server_key_iv: "e4a75714f2ce1945144b2ba01f661bba"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.576658 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7b2c86f5cfbb4d31b27fa7697a91e7cf"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "d880110dd35a12bbab6680139cc03793"
server_key_iv: "e4a75714f2ce1945144b2ba01f661bba"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.580521 8807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:03:41.583010 8822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.584367 8807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:41.584491 8807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "7b2c86f5cfbb4d31b27fa7697a91e7cf"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "d880110dd35a12bbab6680139cc03793"
server_key_iv: "e4a75714f2ce1945144b2ba01f661bba"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.584597 8807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:41.595707 8807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:41.598480 8807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:41.598652 8807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:41.599248 8807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:41.600138 8807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:41.600188 8807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.600242 8807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:41.600266 8807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.608683 8807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33021
I20260924 04:03:41.608704 8935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33021 every 8 connection(s)
I20260924 04:03:41.609619 8807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:03:41 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:41.618806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 8807
I20260924 04:03:41.618947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:41.619252 392 external_mini_cluster.cc:1428] Setting key f2aa3b27f9703891814caa39b6ea1db9
I20260924 04:03:41.622686 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:41.611300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33689 (local address 127.0.98.62:43579)
0924 04:03:41.611451 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:41.611456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:41.612298 (+ 842us) server_negotiation.cc:407] Connection header received
0924 04:03:41.613633 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:41.613636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:41.613682 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:41.613753 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:41.614896 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.615391 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.616530 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.616749 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.616819 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:41.619278 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:41.619281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:41.619283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:41.619312 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:41.620810 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:41.621275 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:41.621279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:41.621281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:41.621354 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:41.621657 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:41.621661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:41.621663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:41.621838 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:41.621928 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:41.622417 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 04:03:41.622533 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:03:41.623754 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:41.611718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:33689)
0924 04:03:41.612161 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:41.612203 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:41.613438 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:41.613909 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:41.613916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:41.614263 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:41.614780 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:41.614793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.615484 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.615487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:41.616428 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:41.616437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.616859 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.616862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:41.617003 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:41.617513 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:41.617528 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:41.619095 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:41.620912 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:41.620918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:41.620935 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:41.621186 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:41.621433 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:41.621436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:41.621438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:41.621575 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:41.621928 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:41.621935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:41.622192 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 04:03:41.622364 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 04:03:41.622723 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260924 04:03:41.624931 8936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43579
I20260924 04:03:41.625168 8936 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:41.625639 8936 heartbeater.cc:511] Master 127.0.98.62:43579 requested a full tablet report, sending...
I20260924 04:03:41.626662 8611 ts_manager.cc:194] Registered new tserver with Master: 7b2c86f5cfbb4d31b27fa7697a91e7cf (127.0.98.2:33021)
I20260924 04:03:41.627240 8611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33689
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:03:41.646939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:41.756701 8941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:41.756980 8941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:41.757023 8941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:41.761173 8941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:41.761233 8941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:41.761291 8941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:41.766637 8941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:41.767633 8941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:41.768668 8941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:41.778254 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:41.778298 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:41.778291 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:41.778614 8941 server_base.cc:1034] running on GCE node
I20260924 04:03:41.779007 8941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:41.779634 8941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:41.780782 8941 hybrid_clock.cc:648] HybridClock initialized: now 1790222621780743 us; error 50 us; skew 500 ppm
Sep 24 04:03:41 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:41.783401 8941 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:03:41.784291 8941 webserver.cc:460] Webserver started at http://127.0.98.3:43611/ using document root <none> and password file <none>
I20260924 04:03:41.784813 8941 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:41.784883 8941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:41.785080 8941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:41.787113 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "6515384f038646448b740195e3e0fefa"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "1e6bbc124bf7053e047b5c7142bd6914"
server_key_iv: "f258e29ef8d5f0c1aca01a3f13af2be4"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.787631 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6515384f038646448b740195e3e0fefa"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "1e6bbc124bf7053e047b5c7142bd6914"
server_key_iv: "f258e29ef8d5f0c1aca01a3f13af2be4"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.791211 8941 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:03:41.793510 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.794764 8941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:03:41.794874 8941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "6515384f038646448b740195e3e0fefa"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "1e6bbc124bf7053e047b5c7142bd6914"
server_key_iv: "f258e29ef8d5f0c1aca01a3f13af2be4"
server_key_version: "encryptionkey@0"
I20260924 04:03:41.794966 8941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:41.828799 8941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:41.832048 8941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:41.832232 8941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:41.832823 8941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:41.833643 8941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:41.833700 8941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.833776 8941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:41.833801 8941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:41.842288 8941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40875
I20260924 04:03:41.842345 9069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40875 every 8 connection(s)
I20260924 04:03:41.843298 8941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:03:41 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:03:41.853497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 8941
I20260924 04:03:41.853622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:41.853888 392 external_mini_cluster.cc:1428] Setting key 3441963861dd2f142e51765b6897433e
I20260924 04:03:41.857221 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:41.845183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59957 (local address 127.0.98.62:43579)
0924 04:03:41.845325 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:41.845329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:41.846219 (+ 890us) server_negotiation.cc:407] Connection header received
0924 04:03:41.847671 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:41.847674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:41.847733 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:41.847804 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:41.849153 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.849681 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.850832 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.851046 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.851133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:41.853602 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:41.853605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:41.853607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:41.853637 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:41.855219 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:41.855786 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:41.855789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:41.855790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:41.855886 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:41.856233 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:41.856236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:41.856237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:41.856430 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:41.856504 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:41.857001 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 04:03:41.857104 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:03:41.858062 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:41.845578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:59957)
0924 04:03:41.846067 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:41.846112 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:41.847456 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:41.848018 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:41.848026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:41.848429 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:41.849008 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:41.849025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.849826 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.849829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:41.850705 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:41.850720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:41.851187 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:41.851191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:41.851354 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:41.851946 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:41.851969 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:41.853421 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:41.855325 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:41.855332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:41.855351 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:41.855701 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:41.855990 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:03:41.855993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:03:41.855995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:03:41.856148 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:03:41.856514 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:41.856519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:41.856779 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 04:03:41.856953 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:03:41.857222 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":112,"threads_started":1}
I20260924 04:03:41.859148 9070 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43579
I20260924 04:03:41.859378 9070 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:41.859876 9070 heartbeater.cc:511] Master 127.0.98.62:43579 requested a full tablet report, sending...
I20260924 04:03:41.860824 8611 ts_manager.cc:194] Registered new tserver with Master: 6515384f038646448b740195e3e0fefa (127.0.98.3:40875)
I20260924 04:03:41.861370 8611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59957
I20260924 04:03:41.867900 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:03:41.940448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 8673
I20260924 04:03:41.945358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 8807
I20260924 04:03:41.951486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 8941
I20260924 04:03:41.956059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 8579
I20260924 04:03:41.961458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43579
--webserver_interface=127.0.98.62
--webserver_port=45565
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:03:42.076339 9079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:42.076699 9079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:42.076781 9079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:42.081015 9079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:42.081099 9079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:42.081126 9079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:42.081147 9079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:42.081173 9079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:42.086843 9079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:42.088042 9079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:42.089123 9079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:42.097880 9087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:42.097898 9084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:42.097990 9079 server_base.cc:1034] running on GCE node
W20260924 04:03:42.097898 9085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:42.098767 9079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:42.099866 9079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:42.101063 9079 hybrid_clock.cc:648] HybridClock initialized: now 1790222622101039 us; error 38 us; skew 500 ppm
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:42.103942 9079 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:03:42.104926 9079 webserver.cc:460] Webserver started at http://127.0.98.62:45565/ using document root <none> and password file <none>
I20260924 04:03:42.105468 9079 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:42.105541 9079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:42.109537 9079 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:42.111894 9094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.113197 9079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:03:42.113308 9079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "ad0d5347e49c466bbdc0b165cb85f15a"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "0b1e084510df819f23496b21b0c3c3bf"
server_key_iv: "5d437681ba110328bb4c669cf854c45a"
server_key_version: "encryptionkey@0"
I20260924 04:03:42.113893 9079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:42.129357 9079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:42.132607 9079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:42.132797 9079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:42.139729 9079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43579
I20260924 04:03:42.139744 9146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43579 every 8 connection(s)
I20260924 04:03:42.140791 9079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:42.144963 9147 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:03:42.146548 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Bootstrap starting.
I20260924 04:03:42.147729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 9079
I20260924 04:03:42.148162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37145
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43579
--webserver_port=38089
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:03:42.162612 9147 log.cc:824] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:42.169306 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:03:42.169651 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Bootstrap complete.
I20260924 04:03:42.173779 9147 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260924 04:03:42.174209 9147 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0d5347e49c466bbdc0b165cb85f15a, State: Initialized, Role: FOLLOWER
I20260924 04:03:42.174701 9147 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260924 04:03:42.174819 9147 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:42.174890 9147 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:42.174983 9147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:03:42.177083 9147 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260924 04:03:42.177423 9147 leader_election.cc:302] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad0d5347e49c466bbdc0b165cb85f15a; no voters:
I20260924 04:03:42.177727 9147 leader_election.cc:288] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:03:42.177835 9151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:03:42.178045 9151 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [term 2 LEADER]: Becoming Leader. State: Replica: ad0d5347e49c466bbdc0b165cb85f15a, State: Running, Role: LEADER
I20260924 04:03:42.178438 9151 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260924 04:03:42.179029 9147 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:42.179813 9152 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } } }
I20260924 04:03:42.179966 9152 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:42.180593 9153 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad0d5347e49c466bbdc0b165cb85f15a. Latest consensus state: current_term: 2 leader_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad0d5347e49c466bbdc0b165cb85f15a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } } }
I20260924 04:03:42.180717 9153 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:42.181061 9160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:42.184445 9160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:42.185104 9160 catalog_manager.cc:1259] Loaded cluster ID: d9cdf4d34a634886b487c9b4b8044b3d
I20260924 04:03:42.185166 9160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:42.187522 9160 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:42.188231 9160 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad0d5347e49c466bbdc0b165cb85f15a: Loaded TSK: 0
I20260924 04:03:42.189459 9160 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:03:42.261240 9149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:42.261543 9149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:42.261605 9149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:42.265825 9149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:42.265905 9149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:42.265995 9149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:42.271557 9149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38089
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:42.272828 9149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:42.273947 9149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:42.284937 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:42.285223 9149 server_base.cc:1034] running on GCE node
W20260924 04:03:42.284957 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:42.284981 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:42.285810 9149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:42.286506 9149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:42.287701 9149 hybrid_clock.cc:648] HybridClock initialized: now 1790222622287668 us; error 44 us; skew 500 ppm
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222622, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:42.290472 9149 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:03:42.291447 9149 webserver.cc:460] Webserver started at http://127.0.98.1:38089/ using document root <none> and password file <none>
I20260924 04:03:42.292021 9149 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:42.292096 9149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:42.296293 9149 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:42.298280 9183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.299513 9149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:42.299610 9149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "8b8d17b1d2db4bd8805737321ee287f9"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "04eedca7395b0feac3b1d5f142d0a245"
server_key_iv: "e445301d9e1c1433900810a38596e79f"
server_key_version: "encryptionkey@0"
I20260924 04:03:42.300186 9149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:42.319618 9149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:42.322781 9149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:42.322954 9149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:42.323536 9149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:42.324393 9149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:42.324447 9149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.324503 9149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:42.324532 9149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.333077 9149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37145
I20260924 04:03:42.333196 9296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37145 every 8 connection(s)
I20260924 04:03:42.334128 9149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:42.334335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 9149
I20260924 04:03:42.334829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33021
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43579
--webserver_port=38571
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:42.347229 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.336614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:53581)
0924 04:03:42.337092 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.337144 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.338403 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.339558 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.339567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.340004 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.340470 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.340484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.341753 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.341756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.342611 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.342620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.343353 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.343357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.343531 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.344088 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.344104 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.346258 (+ 2154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
W20260924 04:03:42.347219 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.336621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53581 (local address 127.0.98.62:43579)
0924 04:03:42.337206 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.337227 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.337278 (+ 51us) server_negotiation.cc:407] Connection header received
0924 04:03:42.338836 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.338848 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.339158 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.339463 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.340643 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.341613 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.342732 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.343202 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.343349 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.346531 (+ 3182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53581: BlockingRecv error: recv got EOF from 127.0.98.1:53581 (error 108)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":122,"threads_started":1}
W20260924 04:03:42.347978 9297 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43579 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43579: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:42.353621 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.348430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:39263)
0924 04:03:42.348564 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.348580 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.348693 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.348930 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.348933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.349114 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.349234 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.349241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.349895 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.349899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.350813 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.350824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.351256 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.351260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.351364 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.351816 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.351832 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.353529 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 04:03:42.354022 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.348532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39263 (local address 127.0.98.62:43579)
0924 04:03:42.348625 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.348628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.348640 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:03:42.348783 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.348786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.348831 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.348895 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.349329 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.349798 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.350945 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.351156 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.351239 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.353693 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39263: BlockingRecv error: recv got EOF from 127.0.98.1:39263 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:42.358870 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.354059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:51211)
0924 04:03:42.354202 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.354214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.354283 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.354553 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.354555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.354665 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.354730 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.354734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.355630 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.355632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.356176 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.356183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.356709 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.356711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.356762 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.357153 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.357164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.358737 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 04:03:42.359004 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.354140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51211 (local address 127.0.98.62:43579)
0924 04:03:42.354271 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.354276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.354293 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:03:42.354393 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.354397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.354442 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.354528 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.354823 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.355531 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.356296 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.356604 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.356719 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.358873 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51211: BlockingRecv error: recv got EOF from 127.0.98.1:51211 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":112,"threads_started":1}
W20260924 04:03:42.359159 9297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:03:42.450192 9299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:42.450482 9299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:42.450555 9299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:42.454612 9299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:42.454684 9299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:42.454766 9299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:42.460232 9299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38571
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:42.461336 9299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:42.462370 9299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:42.471618 9299 server_base.cc:1034] running on GCE node
W20260924 04:03:42.471699 9308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:42.471635 9307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:42.471792 9310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:42.472172 9299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:42.472836 9299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:42.473994 9299 hybrid_clock.cc:648] HybridClock initialized: now 1790222622473983 us; error 26 us; skew 500 ppm
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222622, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:42.476661 9299 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:03:42.477510 9299 webserver.cc:460] Webserver started at http://127.0.98.2:38571/ using document root <none> and password file <none>
I20260924 04:03:42.478060 9299 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:42.478137 9299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:42.482239 9299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 04:03:42.484429 9317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.485702 9299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:42.485831 9299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "7b2c86f5cfbb4d31b27fa7697a91e7cf"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "d880110dd35a12bbab6680139cc03793"
server_key_iv: "e4a75714f2ce1945144b2ba01f661bba"
server_key_version: "encryptionkey@0"
I20260924 04:03:42.486388 9299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:42.532694 9299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:42.535573 9299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:42.535755 9299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:42.536369 9299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:42.537180 9299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:42.537227 9299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.537281 9299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:42.537310 9299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.545835 9299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33021
I20260924 04:03:42.545902 9430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33021 every 8 connection(s)
I20260924 04:03:42.546857 9299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:42.552029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 9299
I20260924 04:03:42.552477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40875
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43579
--webserver_port=43611
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:03:42.557914 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.548647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54929 (local address 127.0.98.62:43579)
0924 04:03:42.548794 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.548799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.549681 (+ 882us) server_negotiation.cc:407] Connection header received
0924 04:03:42.550937 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.550940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.550989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.551061 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.552283 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.553012 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.554102 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.554433 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.554544 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.557809 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54929: BlockingRecv error: recv got EOF from 127.0.98.2:54929 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:03:42.558763 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.549093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:54929)
0924 04:03:42.549533 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.549576 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.550749 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.551221 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.551228 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.551599 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.552146 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.552165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.553121 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.553124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.553991 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.554000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.554566 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.554568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.554708 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.555240 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.555256 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.557693 (+ 2437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
W20260924 04:03:42.559264 9431 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43579 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43579: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:42.564740 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.559666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:58383)
0924 04:03:42.559844 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.559856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.559941 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.560150 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.560152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.560276 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.560376 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.560381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.561333 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.561335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.561864 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.561870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.562375 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.562377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.562442 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.562918 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.562928 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.564649 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260924 04:03:42.564916 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.559780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58383 (local address 127.0.98.62:43579)
0924 04:03:42.559941 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.559945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.559960 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:03:42.560006 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.560009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.560059 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.560143 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.560473 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.561237 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.561962 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.562281 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.562387 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.564805 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58383: BlockingRecv error: recv got EOF from 127.0.98.2:58383 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:42.570101 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.565189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:40373)
0924 04:03:42.565439 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.565452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.565530 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.565716 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.565718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.565841 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.565909 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.565913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.566856 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.566859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.567360 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.567367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.567854 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.567856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.567933 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.568369 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.568379 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.570011 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190}
W20260924 04:03:42.570256 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.565274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40373 (local address 127.0.98.62:43579)
0924 04:03:42.565377 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.565381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.565504 (+ 123us) server_negotiation.cc:407] Connection header received
0924 04:03:42.565589 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.565592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.565632 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.565705 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.565987 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.566764 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.567445 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.567767 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.567860 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.570149 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40373: BlockingRecv error: recv got EOF from 127.0.98.2:40373 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 04:03:42.570295 9431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:03:42.664326 9434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:42.664603 9434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:42.664665 9434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:42.668882 9434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:42.668951 9434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:42.669029 9434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:42.674410 9434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43611
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:42.675484 9434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:42.676510 9434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:42.685945 9439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:42.685995 9440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:42.685995 9442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:42.686187 9434 server_base.cc:1034] running on GCE node
I20260924 04:03:42.686592 9434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:42.687214 9434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:42.688376 9434 hybrid_clock.cc:648] HybridClock initialized: now 1790222622688355 us; error 33 us; skew 500 ppm
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222622, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:03:42.691028 9434 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:03:42.691953 9434 webserver.cc:460] Webserver started at http://127.0.98.3:43611/ using document root <none> and password file <none>
I20260924 04:03:42.692497 9434 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:42.692569 9434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:42.696451 9434 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:42.698416 9449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.699575 9434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:03:42.699688 9434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "6515384f038646448b740195e3e0fefa"
format_stamp: "Formatted at 2026-09-24 04:03:41 on dist-test-slave-zcsk"
server_key: "1e6bbc124bf7053e047b5c7142bd6914"
server_key_iv: "f258e29ef8d5f0c1aca01a3f13af2be4"
server_key_version: "encryptionkey@0"
I20260924 04:03:42.700244 9434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:42.717785 9434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:42.721058 9434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:42.721243 9434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:42.721845 9434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:42.722667 9434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:42.722714 9434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.722769 9434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:42.722797 9434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:42.731413 9434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40875
I20260924 04:03:42.731479 9562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40875 every 8 connection(s)
I20260924 04:03:42.732470 9434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:42.738850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 9434
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:03:42.743673 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.734290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46649 (local address 127.0.98.62:43579)
0924 04:03:42.734432 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.734436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.735313 (+ 877us) server_negotiation.cc:407] Connection header received
0924 04:03:42.736646 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.736649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.736694 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.736771 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.737905 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.738412 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.739831 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.740128 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.740230 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.743572 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46649: BlockingRecv error: recv got EOF from 127.0.98.3:46649 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:03:42.744635 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.734732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:46649)
0924 04:03:42.735160 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.735205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.736454 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.736936 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.736944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.737302 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.737781 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.737805 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.738498 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.738500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.739645 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.739656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.740266 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.740269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.740470 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.741056 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.741076 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.743445 (+ 2369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
W20260924 04:03:42.745112 9563 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43579 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43579: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 04:03:42.748524 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.739997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.0.98.62:43579)
0924 04:03:42.740118 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.740130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.740143 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:42.740171 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.740173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.740219 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.740303 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.741166 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.741705 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.742693 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.742914 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.742991 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.744589 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:42.744595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:42.744610 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:03:42.744637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:42.746504 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:42.746914 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:42.746919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:42.746925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:42.747048 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:03:42.747295 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:03:42.747298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:03:42.747300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:03:42.747638 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:42.747844 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:42.748115 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 04:03:42.748364 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:42.750340 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.745563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:53289)
0924 04:03:42.745723 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.745738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.745813 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.746026 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.746029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.746176 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.746264 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.746269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.746985 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.746989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.747562 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.747570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.747992 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.747996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.748083 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.748507 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.748518 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.750229 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260924 04:03:42.750589 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.745635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53289 (local address 127.0.98.62:43579)
0924 04:03:42.745798 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.745802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.745815 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:42.745897 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.745900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.745937 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.745998 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.746364 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.746889 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.747674 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.747873 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.747984 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.750354 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53289: BlockingRecv error: recv got EOF from 127.0.98.3:53289 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:42 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:42.755244 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:42.750803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:56535)
0924 04:03:42.750904 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:42.750918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:42.750988 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:42.751176 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:42.751179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:42.751296 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:42.751368 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.751372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.752019 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.752022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.752534 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:42.752541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.752992 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.752995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:42.753053 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.753479 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:42.753490 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:42.755137 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 04:03:42.755388 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:42.750870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56535 (local address 127.0.98.62:43579)
0924 04:03:42.750992 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:42.750997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:42.751010 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:42.751043 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:42.751046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:42.751087 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:42.751152 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:42.751448 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.751937 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.752680 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:42.752894 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:42.752979 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:42.755260 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56535: BlockingRecv error: recv got EOF from 127.0.98.3:56535 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 04:03:42.755474 9563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:03:43 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:43 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:43.364897 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:43.359780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:47839)
0924 04:03:43.360060 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:43.360076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:43.360177 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:43.360460 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:43.360464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:43.360687 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:43.360803 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:43.360810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.361534 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.361536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:43.362110 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:43.362116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.362483 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.362485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:43.362541 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:43.363042 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:43.363056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:43.364768 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
W20260924 04:03:43.365015 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:43.359859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47839 (local address 127.0.98.62:43579)
0924 04:03:43.360113 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:43.360117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:43.360133 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:03:43.360289 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:43.360292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:43.360339 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:43.360434 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:43.360914 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.361449 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.362192 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.362407 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.362474 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:43.364900 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47839: BlockingRecv error: recv got EOF from 127.0.98.1:47839 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
Sep 24 04:03:43 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:43 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:43.575757 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:43.570873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:58727)
0924 04:03:43.571110 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:43.571123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:43.571213 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:43.571415 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:43.571417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:43.571580 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:43.571692 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:43.571712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.572438 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.572440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:43.572999 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:43.573005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.573393 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.573395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:43.573454 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:43.573954 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:43.573966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:43.575619 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
W20260924 04:03:43.575973 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:43.570924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58727 (local address 127.0.98.62:43579)
0924 04:03:43.571051 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:43.571055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:43.571184 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:03:43.571282 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:43.571285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:43.571330 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:43.571392 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:43.571793 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.572354 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.573086 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.573310 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.573385 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:43.575814 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58727: BlockingRecv error: recv got EOF from 127.0.98.2:58727 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 04:03:43 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:43 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:43.761462 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:43.756184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:53299)
0924 04:03:43.756450 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:43.756465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:43.756582 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:43.756836 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:43.756839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:43.757061 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:43.757189 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:43.757197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.758027 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.758030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:43.758593 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:43.758601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.759043 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.759045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:43.759108 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:43.759592 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:43.759602 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:43.761331 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
W20260924 04:03:43.761598 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:43.756193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53299 (local address 127.0.98.62:43579)
0924 04:03:43.756307 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:43.756311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:43.756552 (+ 241us) server_negotiation.cc:407] Connection header received
0924 04:03:43.756666 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:43.756668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:43.756723 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:43.756817 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:43.757308 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.757914 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.758723 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:43.758943 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:43.759027 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:43.761477 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53299: BlockingRecv error: recv got EOF from 127.0.98.3:53299 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 04:03:44 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:44 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:44.370779 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:44.365615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:59803)
0924 04:03:44.365859 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:44.365872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:44.365968 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:44.366240 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:44.366243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:44.366436 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:44.366541 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:44.366547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.367271 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.367274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:44.367836 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:44.367844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.368277 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.368279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:44.368338 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:44.368836 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:44.368848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:44.370635 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
W20260924 04:03:44.370924 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:44.365655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59803 (local address 127.0.98.62:43579)
0924 04:03:44.365865 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:44.365869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:44.365930 (+ 61us) server_negotiation.cc:407] Connection header received
0924 04:03:44.366035 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:44.366037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:44.366080 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:44.366161 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:44.366650 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.367169 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.367961 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.368191 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.368269 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:44.370784 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59803: BlockingRecv error: recv got EOF from 127.0.98.1:59803 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
Sep 24 04:03:44 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:44 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:44.582111 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:44.576543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:55035)
0924 04:03:44.576807 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:44.576822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:44.576949 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:44.577220 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:44.577223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:44.577400 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:44.577539 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:44.577545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.578295 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.578298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:44.578912 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:44.578921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.579336 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.579339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:44.579409 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:44.580062 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:44.580078 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:44.581929 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
W20260924 04:03:44.582178 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:44.576689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55035 (local address 127.0.98.62:43579)
0924 04:03:44.576829 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:44.576832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:44.576902 (+ 70us) server_negotiation.cc:407] Connection header received
0924 04:03:44.577037 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:44.577039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:44.577088 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:44.577151 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:44.577642 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.578176 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.579011 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.579226 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.579296 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:44.582080 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55035: BlockingRecv error: recv got EOF from 127.0.98.2:55035 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 04:03:44 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:44 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:44.766985 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:44.762140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:51945)
0924 04:03:44.762398 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:44.762410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:44.762503 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:44.762704 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:44.762706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:44.762872 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:44.762965 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:44.762970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.763634 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.763636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:44.764254 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:44.764262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.764622 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.764624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:44.764683 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:44.765168 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:44.765180 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:44.766854 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
W20260924 04:03:44.767086 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:44.762273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51945 (local address 127.0.98.62:43579)
0924 04:03:44.762401 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:44.762405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:44.762470 (+ 65us) server_negotiation.cc:407] Connection header received
0924 04:03:44.762570 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:44.762573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:44.762618 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:44.762680 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:44.763043 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.763553 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.764337 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:44.764542 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:44.764613 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:44.766978 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51945: BlockingRecv error: recv got EOF from 127.0.98.3:51945 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 04:03:45 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:45 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:45.376775 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:45.371472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:56033)
0924 04:03:45.371803 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:45.371819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:45.371968 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:45.372252 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:45.372255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:45.372446 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:45.372543 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:45.372549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.373301 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.373305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:45.373860 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:45.373868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.374298 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.374302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:45.374382 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:45.374846 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:45.374858 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:45.376596 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":75,"threads_started":1}
W20260924 04:03:45.376855 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:45.371536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56033 (local address 127.0.98.62:43579)
0924 04:03:45.371805 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:45.371810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:45.371924 (+ 114us) server_negotiation.cc:407] Connection header received
0924 04:03:45.372067 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:45.372071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:45.372127 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:45.372211 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:45.372658 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.373195 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.373970 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.374199 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.374284 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:45.376743 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56033: BlockingRecv error: recv got EOF from 127.0.98.1:56033 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
Sep 24 04:03:45 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:45 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:45.588155 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:45.582839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:36659)
0924 04:03:45.583167 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:45.583178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:45.583281 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:45.583521 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:45.583524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:45.583719 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:45.583814 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:45.583820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.584603 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.584607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:45.585168 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:45.585176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.585604 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.585608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:45.585691 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:45.586188 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:45.586201 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:45.588013 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":83,"threads_started":1}
W20260924 04:03:45.588243 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:45.582904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36659 (local address 127.0.98.62:43579)
0924 04:03:45.583072 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:45.583077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:45.583248 (+ 171us) server_negotiation.cc:407] Connection header received
0924 04:03:45.583367 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:45.583370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:45.583416 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:45.583492 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:45.583948 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.584466 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.585282 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.585502 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.585586 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:45.588144 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36659: BlockingRecv error: recv got EOF from 127.0.98.2:36659 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 04:03:45 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:45 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:45.777019 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:45.768008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:57367)
0924 04:03:45.768291 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:45.768306 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:45.768422 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:45.768671 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:45.768673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:45.768866 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:45.768980 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:45.768986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.769722 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.769725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:45.770282 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:45.770289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.770700 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.770704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:45.770784 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:45.774814 (+ 4030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:45.774828 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:45.776870 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
W20260924 04:03:45.777171 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:45.768113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57367 (local address 127.0.98.62:43579)
0924 04:03:45.768245 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:45.768248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:45.768377 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:03:45.768504 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:45.768507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:45.768560 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:45.768633 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:45.769088 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.769622 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.770383 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:45.770606 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:45.770689 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:45.777054 (+ 6365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57367: BlockingRecv error: recv got EOF from 127.0.98.3:57367 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 04:03:46 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:46 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:46.382886 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:46.377554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:43969)
0924 04:03:46.377836 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:46.377850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:46.377952 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:46.378236 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:46.378238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:46.378412 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:46.378507 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:46.378512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.379290 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.379293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:46.380011 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:46.380022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.380428 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.380430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:46.380490 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:46.380994 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:46.381008 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:46.382735 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
W20260924 04:03:46.383106 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:46.377615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43969 (local address 127.0.98.62:43579)
0924 04:03:46.377893 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:46.377896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:46.377912 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:03:46.378051 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:46.378053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:46.378095 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:46.378159 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:46.378605 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.379148 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.380104 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.380323 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.380413 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:46.382878 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43969: BlockingRecv error: recv got EOF from 127.0.98.1:43969 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":123,"threads_started":1}
Sep 24 04:03:46 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:46 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:46.594265 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:46.588921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:52205)
0924 04:03:46.589184 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:46.589200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:46.589306 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:46.589561 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:46.589563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:46.589736 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:46.589873 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:46.589880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.590687 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.590690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:46.591235 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:46.591242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.591665 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.591668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:46.591724 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:46.592240 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:46.592251 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:46.594090 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
W20260924 04:03:46.594465 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:46.589030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52205 (local address 127.0.98.62:43579)
0924 04:03:46.589178 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:46.589182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:46.589256 (+ 74us) server_negotiation.cc:407] Connection header received
0924 04:03:46.589398 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:46.589401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:46.589445 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:46.589503 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:46.590014 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.590595 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.591341 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.591574 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.591659 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:46.594251 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52205: BlockingRecv error: recv got EOF from 127.0.98.2:52205 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 04:03:46 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:46 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:46.782972 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:46.777856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:38761)
0924 04:03:46.778131 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:46.778146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:46.778269 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:46.778499 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:46.778502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:46.778692 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:46.778785 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:46.778791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.779547 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.779550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:46.780140 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:46.780148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.780554 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.780557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:46.780631 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:46.781114 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:46.781127 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:46.782809 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
W20260924 04:03:46.783079 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:46.777964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38761 (local address 127.0.98.62:43579)
0924 04:03:46.778128 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:46.778132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:46.778214 (+ 82us) server_negotiation.cc:407] Connection header received
0924 04:03:46.778353 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:46.778356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:46.778403 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:46.778465 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:46.778890 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.779427 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.780246 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:46.780460 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:46.780543 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:46.782980 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38761: BlockingRecv error: recv got EOF from 127.0.98.3:38761 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 04:03:47 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:47 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:47.389029 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:47.383683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:46331)
0924 04:03:47.384100 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:47.384117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:47.384209 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:47.384418 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:47.384420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:47.384593 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:47.384689 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:47.384694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.385386 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.385388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:47.385974 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:47.385981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.386347 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.386349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:47.386408 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:47.386855 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:47.386866 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:47.388868 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
W20260924 04:03:47.389253 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:47.383785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46331 (local address 127.0.98.62:43579)
0924 04:03:47.384051 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:47.384054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:47.384181 (+ 127us) server_negotiation.cc:407] Connection header received
0924 04:03:47.384272 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:47.384274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:47.384322 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:47.384393 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:47.384771 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.385305 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.386056 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.386265 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.386335 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:47.389114 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46331: BlockingRecv error: recv got EOF from 127.0.98.1:46331 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":118,"threads_started":1}
Sep 24 04:03:47 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:47 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:47.600158 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:47.595048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:41015)
0924 04:03:47.595298 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:47.595310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:47.595392 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:47.595602 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:47.595604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:47.595775 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:47.595875 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:47.595881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.596622 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.596624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:47.597230 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:47.597238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.597622 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.597624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:47.597685 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:47.598227 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:47.598238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:47.600030 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
W20260924 04:03:47.600296 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:47.595219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41015 (local address 127.0.98.62:43579)
0924 04:03:47.595352 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:47.595355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:47.595369 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:47.595455 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:47.595457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:47.595499 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:47.595573 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:47.596007 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.596531 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.597324 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.597539 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.597609 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:47.600176 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41015: BlockingRecv error: recv got EOF from 127.0.98.2:41015 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 04:03:47 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:47 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:47.788983 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:47.783898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:36971)
0924 04:03:47.784168 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:47.784181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:47.784272 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:47.784486 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:47.784488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:47.784659 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:47.784753 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:47.784759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.785513 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.785516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:47.786110 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:47.786118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.786567 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.786569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:47.786626 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:47.787101 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:47.787113 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:47.788849 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
W20260924 04:03:47.789075 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:47.784037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36971 (local address 127.0.98.62:43579)
0924 04:03:47.784218 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:47.784221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:47.784236 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:03:47.784347 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:47.784349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:47.784392 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:47.784461 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:47.784864 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.785416 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.786239 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:47.786474 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:47.786564 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:47.788974 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36971: BlockingRecv error: recv got EOF from 127.0.98.3:36971 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 24 04:03:48 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:48 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:48.396601 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:48.390154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:36477)
0924 04:03:48.390434 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:48.390450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:48.390560 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:48.390929 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:48.390933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:48.391203 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:48.391333 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:48.391342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.392510 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.392514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:48.393324 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:48.393337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.393927 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.393931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:48.394024 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:48.394647 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:48.394660 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:48.396479 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
W20260924 04:03:48.396777 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:48.390294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36477 (local address 127.0.98.62:43579)
0924 04:03:48.390555 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:48.390558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:48.390574 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:03:48.390680 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:48.390683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:48.390751 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:48.390850 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:48.391476 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.392331 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.393462 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.393813 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.393917 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:48.396624 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36477: BlockingRecv error: recv got EOF from 127.0.98.1:36477 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
Sep 24 04:03:48 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:48 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:48.606582 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:48.601126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:37407)
0924 04:03:48.601385 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:48.601398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:48.601534 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:48.601821 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:48.601823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:48.602033 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:48.602149 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:48.602158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.603043 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.603045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:48.603680 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:48.603690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.604110 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.604112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:48.604173 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:48.604699 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:48.604711 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:48.606434 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
W20260924 04:03:48.606678 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:48.601226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37407 (local address 127.0.98.62:43579)
0924 04:03:48.601385 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:48.601389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:48.601468 (+ 79us) server_negotiation.cc:407] Connection header received
0924 04:03:48.601649 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:48.601651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:48.601696 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:48.601784 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:48.602260 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.602941 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.603783 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.604020 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.604106 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:48.606580 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37407: BlockingRecv error: recv got EOF from 127.0.98.2:37407 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 04:03:48 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:48 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:48.795130 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:48.789714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:41601)
0924 04:03:48.790008 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:48.790023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:48.790151 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:48.790449 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:48.790453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:48.790625 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:48.790723 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:48.790728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.791489 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.791492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:48.792091 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:48.792101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.792563 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.792566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:48.792644 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:48.793236 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:48.793248 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:48.795009 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
W20260924 04:03:48.795339 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:48.789782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41601 (local address 127.0.98.62:43579)
0924 04:03:48.789921 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:48.789973 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:48.790088 (+ 115us) server_negotiation.cc:407] Connection header received
0924 04:03:48.790241 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:48.790244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:48.790301 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:48.790397 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:48.790839 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.791378 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.792216 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:48.792441 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:48.792520 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:48.795216 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41601: BlockingRecv error: recv got EOF from 127.0.98.3:41601 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 04:03:49 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:49 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:49.402679 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:49.397321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:46819)
0924 04:03:49.397800 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:49.397812 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:49.397908 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:49.398116 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:49.398119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:49.398285 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:49.398378 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:49.398383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.399222 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.399225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:49.399789 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:49.399797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.400282 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.400285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:49.400352 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:49.400820 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:49.400832 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:49.402522 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":84,"threads_started":1}
W20260924 04:03:49.402837 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:49.397361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46819 (local address 127.0.98.62:43579)
0924 04:03:49.397647 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:49.397651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:49.397870 (+ 219us) server_negotiation.cc:407] Connection header received
0924 04:03:49.397972 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:49.397974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:49.398022 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:49.398095 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:49.398506 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.399084 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.399931 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.400182 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.400261 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:49.402694 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46819: BlockingRecv error: recv got EOF from 127.0.98.1:46819 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
Sep 24 04:03:49 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:49 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:49.612519 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:49.607233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:60589)
0924 04:03:49.607511 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:49.607523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:49.607622 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:49.607871 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:49.607873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:49.608103 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:49.608202 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:49.608209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.608979 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.608981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:49.609561 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:49.609568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.610015 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.610017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:49.610093 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:49.610606 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:49.610618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:49.612399 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":64,"threads_started":1}
W20260924 04:03:49.612656 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:49.607343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60589 (local address 127.0.98.62:43579)
0924 04:03:49.607480 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:49.607483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:49.607588 (+ 105us) server_negotiation.cc:407] Connection header received
0924 04:03:49.607729 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:49.607732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:49.607780 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:49.607848 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:49.608336 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.608860 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.609674 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.609920 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.610002 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:49.612546 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60589: BlockingRecv error: recv got EOF from 127.0.98.2:60589 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 04:03:49 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:49 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:49.800871 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:49.795849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:46577)
0924 04:03:49.796177 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:49.796189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:49.796279 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:49.796472 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:49.796474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:49.796640 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:49.796734 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:49.796739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.797465 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.797469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:49.798057 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:49.798066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.798490 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.798494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:49.798567 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:49.799011 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:49.799022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:49.800755 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":92,"threads_started":1}
W20260924 04:03:49.801028 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:49.795950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46577 (local address 127.0.98.62:43579)
0924 04:03:49.796071 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:49.796075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:49.796233 (+ 158us) server_negotiation.cc:407] Connection header received
0924 04:03:49.796339 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:49.796341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:49.796385 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:49.796447 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:49.796815 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.797357 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.798157 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:49.798371 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:49.798443 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:49.800898 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46577: BlockingRecv error: recv got EOF from 127.0.98.3:46577 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 04:03:50 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:50 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:50.408803 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:50.403996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:51043)
0924 04:03:50.404231 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:50.404246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:50.404333 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:50.404548 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:50.404550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:50.404714 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:50.404810 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:50.404815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.405558 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.405560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:50.406128 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:50.406135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.406530 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.406532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:50.406585 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:50.407025 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:50.407036 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:50.408686 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
W20260924 04:03:50.408963 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:50.404108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51043 (local address 127.0.98.62:43579)
0924 04:03:50.404326 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:50.404329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:50.404343 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:50.404398 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:50.404400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:50.404441 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:50.404522 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:50.404924 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.405475 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.406215 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.406448 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.406522 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:50.408827 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51043: BlockingRecv error: recv got EOF from 127.0.98.1:51043 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
Sep 24 04:03:50 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:50 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:50.618705 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:50.613243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:40141)
0924 04:03:50.613502 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:50.613514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:50.613623 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:50.613853 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:50.613856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:50.614054 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:50.614147 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:50.614153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.615030 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.615033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:50.615587 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:50.615595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.616085 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.616089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:50.616167 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:50.616646 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:50.616658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:50.618532 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
W20260924 04:03:50.618836 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:50.613321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40141 (local address 127.0.98.62:43579)
0924 04:03:50.613459 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:50.613463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:50.613579 (+ 116us) server_negotiation.cc:407] Connection header received
0924 04:03:50.613715 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:50.613718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:50.613761 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:50.613821 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:50.614258 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.614926 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.615691 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.615988 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.616079 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:50.618687 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40141: BlockingRecv error: recv got EOF from 127.0.98.2:40141 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 04:03:50 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:50 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:50.810142 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:50.801577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:37315)
0924 04:03:50.801812 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:50.801825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:50.801928 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:50.802142 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:50.802144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:50.802336 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:50.802431 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:50.802436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.803197 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.803200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:50.803798 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:50.803808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.804332 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.804335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:50.804405 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:50.807876 (+ 3471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:50.807891 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:50.809991 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
W20260924 04:03:50.810225 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:50.801660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37315 (local address 127.0.98.62:43579)
0924 04:03:50.801806 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:50.801810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:50.801891 (+ 81us) server_negotiation.cc:407] Connection header received
0924 04:03:50.802004 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:50.802006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:50.802052 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:50.802116 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:50.802513 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.803059 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.803939 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:50.804178 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:50.804256 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:50.810129 (+ 5873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37315: BlockingRecv error: recv got EOF from 127.0.98.3:37315 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 04:03:51 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:51 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:51.414983 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:51.409607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:49985)
0924 04:03:51.409861 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:51.409874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:51.409968 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:51.410194 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:51.410196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:51.410364 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:51.410458 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:51.410463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.411223 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.411225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:51.411976 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:51.411989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.412494 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.412498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:51.412576 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:51.413098 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:51.413113 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:51.414832 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
W20260924 04:03:51.415098 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:51.409727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49985 (local address 127.0.98.62:43579)
0924 04:03:51.409955 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:51.409958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:51.409974 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:03:51.410044 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:51.410046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:51.410095 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:51.410181 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:51.410581 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.411130 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.412098 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.412388 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.412473 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:51.414978 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49985: BlockingRecv error: recv got EOF from 127.0.98.1:49985 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
Sep 24 04:03:51 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:51 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:51.625053 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:51.619519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:40963)
0924 04:03:51.619792 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:51.619808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:51.619957 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:51.620190 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:51.620193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:51.620400 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:51.620493 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:51.620499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.621316 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.621319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:51.621876 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:51.621885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.622314 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.622318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:51.622399 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:51.622924 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:51.622936 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:51.624941 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
W20260924 04:03:51.625211 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:51.619614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40963 (local address 127.0.98.62:43579)
0924 04:03:51.619813 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:51.619817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:51.619880 (+ 63us) server_negotiation.cc:407] Connection header received
0924 04:03:51.620043 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:51.620046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:51.620093 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:51.620155 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:51.620604 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.621207 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.621981 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.622200 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.622283 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:51.625088 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40963: BlockingRecv error: recv got EOF from 127.0.98.2:40963 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 04:03:51 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:51 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:51.816504 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:51.810886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:33161)
0924 04:03:51.811171 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:51.811186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:51.811301 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:51.811571 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:51.811574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:51.811850 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:51.812024 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:51.812033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.812968 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.812971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:51.813538 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:51.813546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.813960 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.813963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:51.814027 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:51.814512 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:51.814524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:51.816386 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
W20260924 04:03:51.816604 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:51.811011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33161 (local address 127.0.98.62:43579)
0924 04:03:51.811152 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:51.811156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:51.811261 (+ 105us) server_negotiation.cc:407] Connection header received
0924 04:03:51.811395 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:51.811398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:51.811453 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:51.811518 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:51.812140 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.812861 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.813640 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:51.813871 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:51.813943 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:51.816510 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33161: BlockingRecv error: recv got EOF from 127.0.98.3:33161 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 04:03:52 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:52 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:52.420939 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:52.415814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:35439)
0924 04:03:52.416089 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:52.416102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:52.416187 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:52.416405 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:52.416407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:52.416572 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:52.416667 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:52.416672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.417568 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.417570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:52.418138 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:52.418146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.418612 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.418615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:52.418677 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:52.419135 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:52.419145 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:52.420823 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
W20260924 04:03:52.421072 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:52.415956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35439 (local address 127.0.98.62:43579)
0924 04:03:52.416179 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:52.416182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:52.416206 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:03:52.416265 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:52.416267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:52.416310 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:52.416384 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:52.416808 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.417483 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.418256 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.418499 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.418578 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:52.420952 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35439: BlockingRecv error: recv got EOF from 127.0.98.1:35439 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":61,"threads_started":1}
Sep 24 04:03:52 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:52 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:52.631142 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:52.625818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:37887)
0924 04:03:52.626097 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:52.626111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:52.626210 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:52.626412 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:52.626414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:52.626603 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:52.626700 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:52.626706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.627487 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.627490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:52.628102 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:52.628111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.628547 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.628550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:52.628636 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:52.629104 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:52.629114 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:52.630999 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":70,"threads_started":1}
W20260924 04:03:52.631222 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:52.625907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37887 (local address 127.0.98.62:43579)
0924 04:03:52.626069 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:52.626073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:52.626227 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:03:52.626273 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:52.626275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:52.626320 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:52.626388 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:52.626834 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.627395 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.628214 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.628439 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.628514 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:52.631124 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37887: BlockingRecv error: recv got EOF from 127.0.98.2:37887 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 04:03:52 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:52 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:52.822621 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:52.817330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:48503)
0924 04:03:52.817580 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:52.817593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:52.817699 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:52.817905 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:52.817908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:52.818082 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:52.818195 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:52.818201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.818981 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.818984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:52.819565 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:52.819573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.820055 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.820059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:52.820138 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:52.820692 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:52.820705 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:52.822504 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
W20260924 04:03:52.822744 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:52.817457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48503 (local address 127.0.98.62:43579)
0924 04:03:52.817579 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:52.817582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:52.817648 (+ 66us) server_negotiation.cc:407] Connection header received
0924 04:03:52.817765 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:52.817767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:52.817811 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:52.817881 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:52.818285 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.818865 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.819652 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:52.819890 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:52.820012 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:52.822634 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48503: BlockingRecv error: recv got EOF from 127.0.98.3:48503 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 04:03:53 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:53 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:53.426738 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:53.421686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:48801)
0924 04:03:53.421940 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:53.421965 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:53.422052 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:53.422262 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:53.422264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:53.422431 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:53.422526 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:53.422532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.423244 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.423246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:53.423815 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:53.423824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.424348 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.424353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:53.424436 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:53.424902 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:53.424913 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:53.426601 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
W20260924 04:03:53.426865 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:53.421770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48801 (local address 127.0.98.62:43579)
0924 04:03:53.422005 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:53.422008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:53.422021 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:03:53.422119 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:53.422121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:53.422161 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:53.422238 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:53.422626 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.423147 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.423954 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.424189 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.424268 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:53.426736 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48801: BlockingRecv error: recv got EOF from 127.0.98.1:48801 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
Sep 24 04:03:53 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:53 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:53.637364 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:53.632055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:40207)
0924 04:03:53.632346 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:53.632359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:53.632482 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:53.632782 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:53.632784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:53.632991 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:53.633114 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:53.633121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.633926 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.633929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:53.634524 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:53.634533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.634939 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.634942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:53.635007 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:53.635503 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:53.635514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:53.637244 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
W20260924 04:03:53.637478 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:53.632129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40207 (local address 127.0.98.62:43579)
0924 04:03:53.632278 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:53.632283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:53.632439 (+ 156us) server_negotiation.cc:407] Connection header received
0924 04:03:53.632602 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:53.632606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:53.632664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:53.632741 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:53.633212 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.633826 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.634622 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.634836 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.634912 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:53.637379 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40207: BlockingRecv error: recv got EOF from 127.0.98.2:40207 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 04:03:53 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:53 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:53.828679 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:53.823369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:58165)
0924 04:03:53.823647 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:53.823661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:53.823762 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:53.824063 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:53.824065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:53.824235 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:53.824328 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:53.824333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.825128 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.825131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:53.825717 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:53.825727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.826206 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.826209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:53.826278 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:53.826799 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:53.826811 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:53.828561 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
W20260924 04:03:53.828811 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:53.823420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58165 (local address 127.0.98.62:43579)
0924 04:03:53.823586 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:53.823591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:53.823740 (+ 149us) server_negotiation.cc:407] Connection header received
0924 04:03:53.823889 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:53.823892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:53.823965 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:53.824039 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:53.824427 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.825000 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.825846 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:53.826092 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:53.826180 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:53.828710 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58165: BlockingRecv error: recv got EOF from 127.0.98.3:58165 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 04:03:54 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:54 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:54.432996 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:54.427666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:43019)
0924 04:03:54.427972 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:54.427986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:54.428078 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:54.428380 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:54.428383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:54.428562 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:54.428663 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:54.428668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.429390 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.429393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:54.429975 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:54.429984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.430429 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.430433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:54.430505 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:54.430989 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:54.431001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:54.432840 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
W20260924 04:03:54.433146 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:54.427784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43019 (local address 127.0.98.62:43579)
0924 04:03:54.428030 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:54.428033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:54.428047 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:03:54.428205 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:54.428207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:54.428255 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:54.428337 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:54.428759 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.429275 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.430100 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.430328 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.430405 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:54.433006 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43019: BlockingRecv error: recv got EOF from 127.0.98.1:43019 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
Sep 24 04:03:54 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:54 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:54.643599 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:54.638135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:42873)
0924 04:03:54.638379 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:54.638394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:54.638503 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:54.638761 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:54.638765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:54.639005 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:54.639124 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:54.639132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.640047 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.640049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:54.640640 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:54.640649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.641068 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.641070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:54.641130 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:54.641607 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:54.641622 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:54.643482 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
W20260924 04:03:54.643736 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:54.638262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42873 (local address 127.0.98.62:43579)
0924 04:03:54.638387 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:54.638391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:54.638470 (+ 79us) server_negotiation.cc:407] Connection header received
0924 04:03:54.638576 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:54.638578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:54.638622 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:54.638708 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:54.639246 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.639940 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.640752 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.640978 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.641058 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:54.643630 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42873: BlockingRecv error: recv got EOF from 127.0.98.2:42873 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 04:03:54 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:54 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:54.834717 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:54.829445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:39915)
0924 04:03:54.829687 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:54.829699 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:54.829790 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:54.830074 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:54.830077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:54.830256 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:54.830360 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:54.830366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.831069 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.831072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:54.831663 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:54.831672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.832101 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.832104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:54.832177 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:54.832720 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:54.832731 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:54.834568 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
W20260924 04:03:54.834805 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:54.829531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39915 (local address 127.0.98.62:43579)
0924 04:03:54.829643 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:54.829647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:54.829756 (+ 109us) server_negotiation.cc:407] Connection header received
0924 04:03:54.829853 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:54.829855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:54.829899 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:54.829966 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:54.830465 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.830978 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.831768 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:54.832011 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:54.832084 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:54.834708 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39915: BlockingRecv error: recv got EOF from 127.0.98.3:39915 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 04:03:55 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:55 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:55.439637 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:55.434262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:46109)
0924 04:03:55.434514 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:55.434528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:55.434627 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:55.434862 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:55.434864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:55.435076 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:55.435193 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:55.435200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.436145 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.436148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:55.436702 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:55.436710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.437132 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.437134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:55.437192 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:55.437626 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:55.437637 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:55.439516 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
W20260924 04:03:55.439785 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:55.434346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46109 (local address 127.0.98.62:43579)
0924 04:03:55.434613 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:55.434618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:55.434636 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:03:55.434693 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:55.434696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:55.434747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:55.434863 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:55.435310 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.436040 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.436803 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.437047 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.437123 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:55.439653 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46109: BlockingRecv error: recv got EOF from 127.0.98.1:46109 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
Sep 24 04:03:55 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:55 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:55.650729 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:55.644483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:51901)
0924 04:03:55.644839 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:55.644854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:55.644963 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:55.645181 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:55.645183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:55.645426 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:55.645555 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:55.645562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.646649 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.646652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:55.647380 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:55.647391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.647964 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.647969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:55.648065 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:55.648637 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:55.648652 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:55.650577 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":72,"threads_started":1}
W20260924 04:03:55.650816 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:55.644744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51901 (local address 127.0.98.62:43579)
0924 04:03:55.644867 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:55.644870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:55.644913 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:03:55.645039 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:55.645042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:55.645086 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:55.645153 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:55.645684 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.646524 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.647487 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.647788 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.647954 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:55.650715 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51901: BlockingRecv error: recv got EOF from 127.0.98.2:51901 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 04:03:55 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:55 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:55.844924 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:55.835678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:36723)
0924 04:03:55.835986 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:55.836000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:55.836106 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:55.836393 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:55.836396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:55.836584 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:55.836697 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:55.836704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.837664 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.837667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:55.838231 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:55.838240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.838729 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.838732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:55.838798 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:55.842663 (+ 3865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:55.842679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:55.844764 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
W20260924 04:03:55.845129 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:55.835800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36723 (local address 127.0.98.62:43579)
0924 04:03:55.835997 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:55.836002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:55.836076 (+ 74us) server_negotiation.cc:407] Connection header received
0924 04:03:55.836196 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:55.836200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:55.836256 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:55.836334 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:55.836821 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.837558 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.838389 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:55.838617 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:55.838706 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:55.844991 (+ 6285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36723: BlockingRecv error: recv got EOF from 127.0.98.3:36723 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 04:03:56 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:56 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:56.446190 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:56.440368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:47861)
0924 04:03:56.440635 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:56.440648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:56.440736 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:56.441010 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:56.441013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:56.441225 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:56.441340 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:56.441347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.442132 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.442137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:56.442722 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:56.442731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.443226 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.443229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:56.443300 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:56.443973 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:56.443988 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:56.446019 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
W20260924 04:03:56.446352 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:56.440433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47861 (local address 127.0.98.62:43579)
0924 04:03:56.440709 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:56.440712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:56.440724 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:03:56.440822 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:56.440824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:56.440864 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:56.440936 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:56.441482 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.442025 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.442854 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.443082 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.443214 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:56.446230 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47861: BlockingRecv error: recv got EOF from 127.0.98.1:47861 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":54,"threads_started":1}
Sep 24 04:03:56 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:56 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:56.656909 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:56.651666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:58809)
0924 04:03:56.651950 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:56.651964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:56.652085 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:56.652349 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:56.652352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:56.652540 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:56.652643 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:56.652649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.653391 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.653394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:56.653985 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:56.653994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.654449 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.654451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:56.654513 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:56.654969 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:56.654980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:56.656755 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
W20260924 04:03:56.657037 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:56.651731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58809 (local address 127.0.98.62:43579)
0924 04:03:56.651875 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:56.651879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:56.652038 (+ 159us) server_negotiation.cc:407] Connection header received
0924 04:03:56.652168 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:56.652171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:56.652228 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:56.652338 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:56.652730 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.653284 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.654115 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.654345 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.654430 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:56.656916 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58809: BlockingRecv error: recv got EOF from 127.0.98.2:58809 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 04:03:56 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:56 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:56.851344 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:56.845815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:60671)
0924 04:03:56.846094 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:56.846107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:56.846238 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:56.846520 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:56.846523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:56.846714 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:56.846810 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:56.846815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.847594 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.847597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:56.848243 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:56.848253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.848682 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.848685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:56.848758 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:56.849250 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:56.849262 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:56.851131 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
W20260924 04:03:56.851366 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:56.845937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60671 (local address 127.0.98.62:43579)
0924 04:03:56.846082 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:56.846087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:56.846173 (+ 86us) server_negotiation.cc:407] Connection header received
0924 04:03:56.846316 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:56.846318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:56.846367 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:56.846445 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:56.846947 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.847481 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.848353 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:56.848575 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:56.848658 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:56.851269 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60671: BlockingRecv error: recv got EOF from 127.0.98.3:60671 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 04:03:57 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:57 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:57.452490 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:57.447148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:52855)
0924 04:03:57.447497 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:57.447513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:57.447610 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:57.447842 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:57.447844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:57.448071 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:57.448167 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:57.448173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.448880 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.448883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:57.449496 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:57.449504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.449976 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.449979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:57.450081 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:57.450598 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:57.450609 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:57.452365 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":75,"threads_started":1}
W20260924 04:03:57.452651 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:57.447204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52855 (local address 127.0.98.62:43579)
0924 04:03:57.447450 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:57.447455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:57.447578 (+ 123us) server_negotiation.cc:407] Connection header received
0924 04:03:57.447678 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:57.447681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:57.447724 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:57.447799 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:57.448257 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.448777 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.449628 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.449858 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.449953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:57.452522 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52855: BlockingRecv error: recv got EOF from 127.0.98.1:52855 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
Sep 24 04:03:57 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:03:57 dist-test-slave-zcsk krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:03:57.663159 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:57.657802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:44041)
0924 04:03:57.658047 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:57.658061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:57.658158 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:57.658412 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:57.658414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:57.658602 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:57.658718 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:57.658727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.659709 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.659711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:57.660337 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:57.660348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.660798 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.660801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:57.660878 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:57.661327 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:03:57.661337 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:57.662998 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43579: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
W20260924 04:03:57.663247 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:57.657901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44041 (local address 127.0.98.62:43579)
0924 04:03:57.658108 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:57.658111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:57.658128 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:03:57.658247 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:57.658251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:57.658306 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:57.658395 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:57.658834 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.659589 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.660455 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.660685 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.660786 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:57.663148 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44041: BlockingRecv error: recv got EOF from 127.0.98.2:44041 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 04:03:57.740309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 9149
I20260924 04:03:57.745271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 9299
I20260924 04:03:57.749980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 9434
I20260924 04:03:57.754532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 9079
2026-09-24T04:03:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17881 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T04:03:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:57Z Disabled control of system clock
I20260924 04:03:57.783267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37901
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:03:57.893633 9629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:57.893934 9629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:57.893970 9629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:57.897982 9629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:57.898054 9629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:57.898068 9629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:57.898083 9629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:57.898097 9629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:57.902884 9629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:57.903976 9629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:57.905310 9629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:57.913661 9637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:57.913650 9635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:57.913724 9629 server_base.cc:1034] running on GCE node
W20260924 04:03:57.913650 9634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:57.914477 9629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:57.915643 9629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:57.916798 9629 hybrid_clock.cc:648] HybridClock initialized: now 1790222637916773 us; error 41 us; skew 500 ppm
I20260924 04:03:57.918732 9629 webserver.cc:460] Webserver started at http://127.0.98.62:43761/ using document root <none> and password file <none>
I20260924 04:03:57.919309 9629 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:57.919375 9629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:57.919580 9629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:57.921692 9629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "862c49383c9e4e88809ea69f73c10956"
format_stamp: "Formatted at 2026-09-24 04:03:57 on dist-test-slave-zcsk"
server_key: "41f0546121e1620f8cfc8ae0e5e2ce29"
server_key_iv: "7968e2fbf2270b7574d8a8fac8927203"
server_key_version: "encryptionkey@0"
I20260924 04:03:57.922246 9629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "862c49383c9e4e88809ea69f73c10956"
format_stamp: "Formatted at 2026-09-24 04:03:57 on dist-test-slave-zcsk"
server_key: "41f0546121e1620f8cfc8ae0e5e2ce29"
server_key_iv: "7968e2fbf2270b7574d8a8fac8927203"
server_key_version: "encryptionkey@0"
I20260924 04:03:57.926075 9629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:03:57.928507 9643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:57.929785 9629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:57.929889 9629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "862c49383c9e4e88809ea69f73c10956"
format_stamp: "Formatted at 2026-09-24 04:03:57 on dist-test-slave-zcsk"
server_key: "41f0546121e1620f8cfc8ae0e5e2ce29"
server_key_iv: "7968e2fbf2270b7574d8a8fac8927203"
server_key_version: "encryptionkey@0"
I20260924 04:03:57.930008 9629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:57.953866 9629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:57.954592 9629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:57.954775 9629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:57.961539 9629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37901
I20260924 04:03:57.961594 9695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37901 every 8 connection(s)
I20260924 04:03:57.962451 9629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:57.965843 9696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:57.969722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 9629
I20260924 04:03:57.969815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:57.970068 392 external_mini_cluster.cc:1428] Setting key 6bda7e4b0bcb4825a6d6a0cacfc8e403
I20260924 04:03:57.971616 9696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956: Bootstrap starting.
I20260924 04:03:57.973875 9696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:57.974618 9696 log.cc:824] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:57.976907 9696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956: No bootstrap required, opened a new log
I20260924 04:03:57.979698 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:57.971437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.0.98.62:37901)
0924 04:03:57.972220 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:57.972234 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:57.972271 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:03:57.972895 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:57.972913 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:57.973370 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:57.973696 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:57.974137 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.975236 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.976461 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:57.976961 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:57.977154 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:57.977933 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:57.977938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:57.977951 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:57.978371 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:57.978427 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:57.978447 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:57.978526 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:57.978669 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 04:03:57.978934 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":149,"threads_started":1}
I20260924 04:03:57.980687 9696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c49383c9e4e88809ea69f73c10956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37901 } }
I20260924 04:03:57.980916 9696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:57.980971 9696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862c49383c9e4e88809ea69f73c10956, State: Initialized, Role: FOLLOWER
I20260924 04:03:57.981384 9696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c49383c9e4e88809ea69f73c10956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37901 } }
I20260924 04:03:57.981513 9696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:57.981575 9696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:57.981704 9696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:57.982620 9696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c49383c9e4e88809ea69f73c10956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37901 } }
I20260924 04:03:57.982937 9696 leader_election.cc:302] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862c49383c9e4e88809ea69f73c10956; no voters:
I20260924 04:03:57.983218 9696 leader_election.cc:288] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:57.983363 9701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:57.983566 9701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [term 1 LEADER]: Becoming Leader. State: Replica: 862c49383c9e4e88809ea69f73c10956, State: Running, Role: LEADER
I20260924 04:03:57.983939 9701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c49383c9e4e88809ea69f73c10956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37901 } }
I20260924 04:03:57.984441 9696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:57.985483 9702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "862c49383c9e4e88809ea69f73c10956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c49383c9e4e88809ea69f73c10956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37901 } } }
I20260924 04:03:57.985620 9702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:57.985922 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862c49383c9e4e88809ea69f73c10956. Latest consensus state: current_term: 1 leader_uuid: "862c49383c9e4e88809ea69f73c10956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862c49383c9e4e88809ea69f73c10956" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37901 } } }
I20260924 04:03:57.986001 9710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:57.986075 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956 [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:57.988857 9710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:57.994313 9710 catalog_manager.cc:1347] Generated new cluster ID: 246b343355994f00adb445da6d4b315c
I20260924 04:03:57.994400 9710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:58.003495 9710 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:58.004459 9710 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:58.009526 9710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 862c49383c9e4e88809ea69f73c10956: Generated new TSK 0
I20260924 04:03:58.010210 9710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:03:58.020020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37901
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:03:58.128931 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:58.129186 9720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:58.129222 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:58.133198 9720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:58.133255 9720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:58.133308 9720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:58.138105 9720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:58.139092 9720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:58.140435 9720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:58.149766 9726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:58.149780 9725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:58.149829 9728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:58.149852 9720 server_base.cc:1034] running on GCE node
I20260924 04:03:58.150420 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:58.151051 9720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:58.152222 9720 hybrid_clock.cc:648] HybridClock initialized: now 1790222638152197 us; error 36 us; skew 500 ppm
I20260924 04:03:58.154147 9720 webserver.cc:460] Webserver started at http://127.0.98.1:45703/ using document root <none> and password file <none>
I20260924 04:03:58.154704 9720 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:58.154771 9720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:58.154975 9720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:58.157083 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "cacce728fd1a456cacf71ce2d60942db"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "e0e34e33bb1a18d977f6220c4dcf6191"
server_key_iv: "150ed23587475a15ee2c6f4040c186d1"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.157605 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cacce728fd1a456cacf71ce2d60942db"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "e0e34e33bb1a18d977f6220c4dcf6191"
server_key_iv: "150ed23587475a15ee2c6f4040c186d1"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.161175 9720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:58.163594 9734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.164889 9720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:58.165042 9720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "cacce728fd1a456cacf71ce2d60942db"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "e0e34e33bb1a18d977f6220c4dcf6191"
server_key_iv: "150ed23587475a15ee2c6f4040c186d1"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.165146 9720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:58.181821 9720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:58.182545 9720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:58.182718 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:58.183306 9720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:58.184293 9720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:58.184341 9720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.184396 9720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:58.184422 9720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.192906 9720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40735
I20260924 04:03:58.192934 9847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40735 every 8 connection(s)
I20260924 04:03:58.193800 9720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:58.196357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 9720
I20260924 04:03:58.196467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:58.196787 392 external_mini_cluster.cc:1428] Setting key cac96419913032f35ddc082667e54bbb
I20260924 04:03:58.198884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37901
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:03:58.204203 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:58.195665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35717 (local address 127.0.98.62:37901)
0924 04:03:58.195830 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:58.195835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:58.197152 (+ 1317us) server_negotiation.cc:407] Connection header received
0924 04:03:58.198737 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:58.198741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:58.198788 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:58.198863 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:58.200045 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.200578 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.202085 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.202317 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.202405 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:58.203680 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:58.203683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:58.203685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:58.203698 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:58.203718 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:58.203726 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:58.203785 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:58.203992 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 04:03:58.204085 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:03:58.205428 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:58.196104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37901 (local address 127.0.98.1:35717)
0924 04:03:58.196962 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:58.197021 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:58.198473 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:58.199081 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:58.199090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:58.199200 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:58.199879 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:58.199896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.200712 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.200714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:58.201914 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:58.201925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.202482 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.202486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:58.202712 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:58.203350 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:58.203352 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:58.203412 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:58.203508 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:58.203806 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:58.203816 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:58.203821 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:03:58.203995 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:03:58.204328 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":110,"threads_started":1}
I20260924 04:03:58.206930 9848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37901
I20260924 04:03:58.207168 9848 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:58.207705 9848 heartbeater.cc:511] Master 127.0.98.62:37901 requested a full tablet report, sending...
I20260924 04:03:58.209460 9659 ts_manager.cc:194] Registered new tserver with Master: cacce728fd1a456cacf71ce2d60942db (127.0.98.1:40735)
I20260924 04:03:58.211042 9659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35717
W20260924 04:03:58.311736 9851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:58.312053 9851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:58.312098 9851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:58.316154 9851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:58.316211 9851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:58.316265 9851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:58.321023 9851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:58.322077 9851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:58.323428 9851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:58.333349 9851 server_base.cc:1034] running on GCE node
W20260924 04:03:58.333309 9856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:58.333282 9859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:58.333611 9857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:58.334055 9851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:58.334792 9851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:58.335960 9851 hybrid_clock.cc:648] HybridClock initialized: now 1790222638335941 us; error 33 us; skew 500 ppm
I20260924 04:03:58.337868 9851 webserver.cc:460] Webserver started at http://127.0.98.2:40061/ using document root <none> and password file <none>
I20260924 04:03:58.338443 9851 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:58.338511 9851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:58.338717 9851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:58.340845 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "712bdc3eb8e24042bc2dfabe40a617c3"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "649691e2aaf163021c67e5ec454a32ea"
server_key_iv: "151e6bf2b30d36d1ee7b4a8aa198a1d3"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.341378 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "712bdc3eb8e24042bc2dfabe40a617c3"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "649691e2aaf163021c67e5ec454a32ea"
server_key_iv: "151e6bf2b30d36d1ee7b4a8aa198a1d3"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.345078 9851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:58.347615 9865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.348857 9851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:58.349009 9851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "712bdc3eb8e24042bc2dfabe40a617c3"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "649691e2aaf163021c67e5ec454a32ea"
server_key_iv: "151e6bf2b30d36d1ee7b4a8aa198a1d3"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.349110 9851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:58.366794 9851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:58.367515 9851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:58.367689 9851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:58.368407 9851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:58.369408 9851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:58.369455 9851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.369508 9851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:58.369536 9851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.378134 9851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38283
I20260924 04:03:58.378166 9978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38283 every 8 connection(s)
I20260924 04:03:58.379225 9851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:58.385687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 9851
I20260924 04:03:58.385812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:58.386090 392 external_mini_cluster.cc:1428] Setting key 4ebcbbc880db4928364dcfc66f6018c0
I20260924 04:03:58.388322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37901
--builtin_ntp_servers=127.0.98.20:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:03:58.389439 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:58.381091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50707 (local address 127.0.98.62:37901)
0924 04:03:58.381266 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:58.381271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:58.382431 (+ 1160us) server_negotiation.cc:407] Connection header received
0924 04:03:58.383774 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:58.383777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:58.383828 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:58.383922 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:58.384962 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.385717 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.387113 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.387400 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.387509 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:58.388957 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:58.388960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:58.388962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:58.388974 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:58.389032 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:58.389041 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:58.389110 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:58.389269 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 04:03:58.389321 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:03:58.390483 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:58.381554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37901 (local address 127.0.98.2:50707)
0924 04:03:58.382246 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:58.382286 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:58.383541 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:58.384087 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:58.384094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:58.384195 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:58.384740 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:58.384754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.385879 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.385883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:58.386989 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:58.386999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.387587 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.387595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:58.387805 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:58.388628 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:58.388630 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:58.388691 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:58.388785 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:58.389127 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:58.389134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:58.389145 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 04:03:58.389270 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:03:58.389597 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260924 04:03:58.391644 9979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37901
I20260924 04:03:58.391942 9979 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:58.392618 9979 heartbeater.cc:511] Master 127.0.98.62:37901 requested a full tablet report, sending...
I20260924 04:03:58.393800 9659 ts_manager.cc:194] Registered new tserver with Master: 712bdc3eb8e24042bc2dfabe40a617c3 (127.0.98.2:38283)
I20260924 04:03:58.394361 9659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50707
W20260924 04:03:58.499557 9982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:58.499825 9982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:58.499861 9982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:58.503858 9982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:58.503939 9982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:58.504002 9982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:58.508685 9982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:58.509826 9982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:58.511206 9982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:58.520977 9987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:58.521152 9988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:58.520977 9990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:58.521281 9982 server_base.cc:1034] running on GCE node
I20260924 04:03:58.521827 9982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:58.522485 9982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:58.523692 9982 hybrid_clock.cc:648] HybridClock initialized: now 1790222638523665 us; error 37 us; skew 500 ppm
I20260924 04:03:58.525691 9982 webserver.cc:460] Webserver started at http://127.0.98.3:43861/ using document root <none> and password file <none>
I20260924 04:03:58.526278 9982 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:58.526342 9982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:58.526553 9982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:58.528752 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a40f9d9baf449c1bd7a5dbb0827521d"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "7be9d2e7165554cd409fc7d6053dc4bb"
server_key_iv: "9912bad60e265956ea52e3e1b5899cbf"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.529320 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a40f9d9baf449c1bd7a5dbb0827521d"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "7be9d2e7165554cd409fc7d6053dc4bb"
server_key_iv: "9912bad60e265956ea52e3e1b5899cbf"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.533179 9982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:03:58.535961 9996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.537281 9982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:58.537392 9982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "2a40f9d9baf449c1bd7a5dbb0827521d"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "7be9d2e7165554cd409fc7d6053dc4bb"
server_key_iv: "9912bad60e265956ea52e3e1b5899cbf"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.537501 9982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:58.597721 9982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:58.598412 9982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:58.598582 9982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:58.599170 9982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:58.600179 9982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:58.600229 9982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.600283 9982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:58.600307 9982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.608716 9982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41279
I20260924 04:03:58.608736 10109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41279 every 8 connection(s)
I20260924 04:03:58.609611 9982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:58.615221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 9982
I20260924 04:03:58.615306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:58.615532 392 external_mini_cluster.cc:1428] Setting key 51c3f8cd3c7f7ee76ab5edfc2f17ee91
I20260924 04:03:58.618618 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:58.611414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54591 (local address 127.0.98.62:37901)
0924 04:03:58.611588 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:58.611592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:58.612654 (+ 1062us) server_negotiation.cc:407] Connection header received
0924 04:03:58.614043 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:58.614047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:58.614103 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:58.614177 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:58.615070 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.615653 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.616833 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.617057 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.617145 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:58.618179 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:58.618183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:58.618186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:58.618197 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:58.618215 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:58.618222 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:58.618277 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:58.618409 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:03:58.618467 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:03:58.619505 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:58.611822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37901 (local address 127.0.98.3:54591)
0924 04:03:58.612508 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:58.612555 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:58.613840 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:58.614352 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:58.614359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:58.614449 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:58.614936 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:58.614948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.615765 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.615768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:58.616723 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:03:58.616733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:03:58.617191 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:03:58.617195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:03:58.617371 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:03:58.617915 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:58.617916 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:58.617957 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:58.618043 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:58.618286 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:58.618294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:58.618300 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:03:58.618404 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:03:58.618691 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260924 04:03:58.620559 10110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37901
I20260924 04:03:58.620776 10110 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:58.621284 10110 heartbeater.cc:511] Master 127.0.98.62:37901 requested a full tablet report, sending...
I20260924 04:03:58.622277 9659 ts_manager.cc:194] Registered new tserver with Master: 2a40f9d9baf449c1bd7a5dbb0827521d (127.0.98.3:41279)
I20260924 04:03:58.622826 9659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54591
I20260924 04:03:58.629369 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:03:58.632301 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:58.631260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.0.98.62:37901)
0924 04:03:58.631417 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:58.631421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:58.631517 (+ 96us) server_negotiation.cc:407] Connection header received
0924 04:03:58.631652 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:58.631655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:58.631702 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:58.631774 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:58.632178 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58260: BlockingRecv error: recv got EOF from 127.0.0.1:58260 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:03:58.632433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 9720
I20260924 04:03:58.637790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 9851
I20260924 04:03:58.642539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 9982
I20260924 04:03:58.647336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 9629
2026-09-24T04:03:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (892 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T04:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:03:58Z Disabled control of system clock
I20260924 04:03:58.675527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36599
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:03:58.784116 10121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:58.784415 10121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:58.784452 10121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:58.788477 10121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:03:58.788542 10121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:58.788555 10121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:58.788571 10121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:03:58.788585 10121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:03:58.793376 10121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:58.794404 10121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:58.795688 10121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:58.804103 10129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:58.804116 10126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:58.804198 10121 server_base.cc:1034] running on GCE node
W20260924 04:03:58.804119 10127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:58.804899 10121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:58.806051 10121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:58.807229 10121 hybrid_clock.cc:648] HybridClock initialized: now 1790222638807211 us; error 30 us; skew 500 ppm
I20260924 04:03:58.809207 10121 webserver.cc:460] Webserver started at http://127.0.98.62:39235/ using document root <none> and password file <none>
I20260924 04:03:58.809787 10121 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:58.809854 10121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:58.810084 10121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:58.812240 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "90e3730992cf41419db7b520b6f030ef"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "b0d714c48921b37dd308b6f56a853a05"
server_key_iv: "ff3c19d1deca75273f1d582038230cb6"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.812781 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "90e3730992cf41419db7b520b6f030ef"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "b0d714c48921b37dd308b6f56a853a05"
server_key_iv: "ff3c19d1deca75273f1d582038230cb6"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.816480 10121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:03:58.818998 10135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:58.820351 10121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:58.820480 10121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "90e3730992cf41419db7b520b6f030ef"
format_stamp: "Formatted at 2026-09-24 04:03:58 on dist-test-slave-zcsk"
server_key: "b0d714c48921b37dd308b6f56a853a05"
server_key_iv: "ff3c19d1deca75273f1d582038230cb6"
server_key_version: "encryptionkey@0"
I20260924 04:03:58.820583 10121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:58.825076 10121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:58.825712 10121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:58.825865 10121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:58.832896 10121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36599
I20260924 04:03:58.832891 10187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36599 every 8 connection(s)
I20260924 04:03:58.833904 10121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:03:58.837167 10188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:03:58.841065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 10121
I20260924 04:03:58.841181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:03:58.841465 392 external_mini_cluster.cc:1428] Setting key 9afd3eeea30b9957f9229cdf40af102f
I20260924 04:03:58.843283 10188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef: Bootstrap starting.
I20260924 04:03:58.845573 10188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef: Neither blocks nor log segments found. Creating new log.
I20260924 04:03:58.846367 10188 log.cc:824] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef: Log is configured to *not* fsync() on all Append() calls
I20260924 04:03:58.847136 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:58.842933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.0.98.62:36599)
0924 04:03:58.843626 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:58.843643 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:58.843679 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:03:58.844365 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:58.844382 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:58.844744 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:58.845108 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:58.845877 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:58.845882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:58.845900 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:58.845911 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:58.845964 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:58.845981 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:58.846073 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:58.846200 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:03:58.846431 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":125,"threads_started":1}
I20260924 04:03:58.848466 10188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef: No bootstrap required, opened a new log
I20260924 04:03:58.852129 10188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e3730992cf41419db7b520b6f030ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } }
I20260924 04:03:58.852351 10188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:03:58.852384 10188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e3730992cf41419db7b520b6f030ef, State: Initialized, Role: FOLLOWER
I20260924 04:03:58.852768 10188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e3730992cf41419db7b520b6f030ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } }
I20260924 04:03:58.852885 10188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:03:58.852921 10188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:03:58.852985 10188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:03:58.853878 10188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e3730992cf41419db7b520b6f030ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } }
I20260924 04:03:58.854225 10188 leader_election.cc:302] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e3730992cf41419db7b520b6f030ef; no voters:
I20260924 04:03:58.854569 10188 leader_election.cc:288] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:03:58.854664 10193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:03:58.854861 10193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [term 1 LEADER]: Becoming Leader. State: Replica: 90e3730992cf41419db7b520b6f030ef, State: Running, Role: LEADER
I20260924 04:03:58.855197 10193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e3730992cf41419db7b520b6f030ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } }
I20260924 04:03:58.855700 10188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:03:58.856751 10194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e3730992cf41419db7b520b6f030ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e3730992cf41419db7b520b6f030ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } } }
I20260924 04:03:58.856864 10194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:58.857131 10195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e3730992cf41419db7b520b6f030ef. Latest consensus state: current_term: 1 leader_uuid: "90e3730992cf41419db7b520b6f030ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e3730992cf41419db7b520b6f030ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36599 } } }
I20260924 04:03:58.857254 10195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef [sys.catalog]: This master's current role is: LEADER
I20260924 04:03:58.857277 10202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:03:58.860258 10202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:03:58.865836 10202 catalog_manager.cc:1347] Generated new cluster ID: 136db85ef018406d9e5e4154fac41fbd
I20260924 04:03:58.865918 10202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:03:58.883898 10202 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:03:58.884276 10202 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:03:58.893795 10202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e3730992cf41419db7b520b6f030ef: Generated new TSK 0
I20260924 04:03:58.894541 10202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:03:58.904079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--builtin_ntp_servers=127.0.98.20:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:03:59.012383 10212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:59.012710 10212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:59.012763 10212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:59.016830 10212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:59.016891 10212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:59.016953 10212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:03:59.021705 10212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:59.022774 10212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:59.024312 10212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:59.033874 10218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:59.033943 10217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:59.034037 10212 server_base.cc:1034] running on GCE node
W20260924 04:03:59.033970 10220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:59.034569 10212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:59.035292 10212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:59.036458 10212 hybrid_clock.cc:648] HybridClock initialized: now 1790222639036435 us; error 33 us; skew 500 ppm
I20260924 04:03:59.038326 10212 webserver.cc:460] Webserver started at http://127.0.98.1:43081/ using document root <none> and password file <none>
I20260924 04:03:59.038939 10212 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:59.039006 10212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:59.039201 10212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:59.041353 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "47cbc1ddeb3e4faf9ee28fb0c42b80f4"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "bac13136b6220edcc87be49963140f6e"
server_key_iv: "f5e431f4f55fe2d16352091752844b8a"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.041921 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47cbc1ddeb3e4faf9ee28fb0c42b80f4"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "bac13136b6220edcc87be49963140f6e"
server_key_iv: "f5e431f4f55fe2d16352091752844b8a"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.045694 10212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:03:59.048244 10226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.049595 10212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:03:59.049716 10212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "47cbc1ddeb3e4faf9ee28fb0c42b80f4"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "bac13136b6220edcc87be49963140f6e"
server_key_iv: "f5e431f4f55fe2d16352091752844b8a"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.049825 10212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:59.054536 10212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:59.055172 10212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:59.055336 10212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:59.055869 10212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:59.056898 10212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:59.056945 10212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.057005 10212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:59.057030 10212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.065147 10212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41143
I20260924 04:03:59.065191 10339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41143 every 8 connection(s)
I20260924 04:03:59.066066 10212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:03:59.069779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 10212
I20260924 04:03:59.069911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:03:59.070217 392 external_mini_cluster.cc:1428] Setting key 90eb1b1c9c0824f6e251ceb3493e2544
I20260924 04:03:59.072338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--builtin_ntp_servers=127.0.98.20:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:03:59.072464 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:59.067855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39387 (local address 127.0.98.62:36599)
0924 04:03:59.068125 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:59.068130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:59.069168 (+ 1038us) server_negotiation.cc:407] Connection header received
0924 04:03:59.070748 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:59.070753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:59.070801 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:59.070868 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:59.072063 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:59.072066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:59.072068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:59.072079 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:59.072099 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:59.072107 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:59.072176 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:59.072316 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:03:59.072350 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260924 04:03:59.073731 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:59.068337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36599 (local address 127.0.98.1:39387)
0924 04:03:59.069021 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:59.069063 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:59.070499 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:59.071066 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:59.071074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:59.071169 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:59.071752 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:59.071754 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:59.071808 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:59.071898 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:59.072179 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:59.072187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:59.072193 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:03:59.072367 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:03:59.072706 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":130,"threads_started":1}
I20260924 04:03:59.074872 10340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36599
I20260924 04:03:59.075094 10340 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:59.075506 10340 heartbeater.cc:511] Master 127.0.98.62:36599 requested a full tablet report, sending...
I20260924 04:03:59.077025 10152 ts_manager.cc:194] Registered new tserver with Master: 47cbc1ddeb3e4faf9ee28fb0c42b80f4 (127.0.98.1:41143)
W20260924 04:03:59.180686 10343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:59.180975 10343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:59.181016 10343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:59.185010 10343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:59.185070 10343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:59.185128 10343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:03:59.189826 10343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:59.190883 10343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:59.192269 10343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:03:59.201771 10343 server_base.cc:1034] running on GCE node
W20260924 04:03:59.201730 10348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:59.201792 10351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:59.201730 10349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:59.202337 10343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:59.203008 10343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:59.204176 10343 hybrid_clock.cc:648] HybridClock initialized: now 1790222639204164 us; error 27 us; skew 500 ppm
I20260924 04:03:59.206094 10343 webserver.cc:460] Webserver started at http://127.0.98.2:41445/ using document root <none> and password file <none>
I20260924 04:03:59.206660 10343 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:59.206728 10343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:59.206918 10343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:59.209052 10343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c72b125748a4b708616a1bbf681308b"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "eba6a76d11f68c4a89d2f0cbb1cabdbd"
server_key_iv: "7fdcf4a5026702330813e5a6720214c3"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.209613 10343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c72b125748a4b708616a1bbf681308b"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "eba6a76d11f68c4a89d2f0cbb1cabdbd"
server_key_iv: "7fdcf4a5026702330813e5a6720214c3"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.213408 10343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:03:59.215957 10357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.217213 10343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:59.217319 10343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "7c72b125748a4b708616a1bbf681308b"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "eba6a76d11f68c4a89d2f0cbb1cabdbd"
server_key_iv: "7fdcf4a5026702330813e5a6720214c3"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.217420 10343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:59.222040 10343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:59.222631 10343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:59.222787 10343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:59.223313 10343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:59.224294 10343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:59.224340 10343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.224390 10343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:59.224416 10343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.232556 10343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37517
I20260924 04:03:59.232595 10470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37517 every 8 connection(s)
I20260924 04:03:59.233433 10343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:03:59.238595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 10343
I20260924 04:03:59.238719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:03:59.239046 392 external_mini_cluster.cc:1428] Setting key c18c8d473bdca660a3f8dae19be09797
I20260924 04:03:59.239566 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:59.235144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59695 (local address 127.0.98.62:36599)
0924 04:03:59.235303 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:59.235308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:59.236473 (+ 1165us) server_negotiation.cc:407] Connection header received
0924 04:03:59.237775 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:59.237777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:59.237817 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:59.237893 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:59.239079 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:59.239082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:59.239084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:59.239099 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:59.239126 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:59.239137 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:59.239240 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:59.239371 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 04:03:59.239423 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:03:59.240726 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:59.235621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36599 (local address 127.0.98.2:59695)
0924 04:03:59.236316 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:59.236359 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:59.237563 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:59.238050 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:59.238057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:59.238152 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:59.238734 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:59.238736 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:59.238793 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:59.238891 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:59.239223 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:59.239232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:59.239238 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:03:59.239411 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 04:03:59.239703 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260924 04:03:59.241132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--builtin_ntp_servers=127.0.98.20:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:03:59.241989 10471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36599
I20260924 04:03:59.242209 10471 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:59.242679 10471 heartbeater.cc:511] Master 127.0.98.62:36599 requested a full tablet report, sending...
I20260924 04:03:59.243837 10152 ts_manager.cc:194] Registered new tserver with Master: 7c72b125748a4b708616a1bbf681308b (127.0.98.2:37517)
W20260924 04:03:59.350667 10474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:03:59.350950 10474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:03:59.350988 10474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:03:59.355099 10474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:03:59.355162 10474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:03:59.355223 10474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:03:59.360297 10474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:03:59.361472 10474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:03:59.362803 10474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:03:59.372462 10480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:59.372430 10482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:03:59.372514 10479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:03:59.372756 10474 server_base.cc:1034] running on GCE node
I20260924 04:03:59.373196 10474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:03:59.373884 10474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:03:59.375056 10474 hybrid_clock.cc:648] HybridClock initialized: now 1790222639375044 us; error 27 us; skew 500 ppm
I20260924 04:03:59.377038 10474 webserver.cc:460] Webserver started at http://127.0.98.3:37453/ using document root <none> and password file <none>
I20260924 04:03:59.377607 10474 fs_manager.cc:359] Metadata directory not provided
I20260924 04:03:59.377668 10474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:03:59.377856 10474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:03:59.379953 10474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa4a9e2b74d540569921a4424de4a941"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "78ddfbf860004abeaef52c5b433e03e7"
server_key_iv: "67f2f26b8eb180796b8f8b0681ec9354"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.380513 10474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa4a9e2b74d540569921a4424de4a941"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "78ddfbf860004abeaef52c5b433e03e7"
server_key_iv: "67f2f26b8eb180796b8f8b0681ec9354"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.384248 10474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:03:59.386660 10488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.388082 10474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:03:59.388206 10474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "fa4a9e2b74d540569921a4424de4a941"
format_stamp: "Formatted at 2026-09-24 04:03:59 on dist-test-slave-zcsk"
server_key: "78ddfbf860004abeaef52c5b433e03e7"
server_key_iv: "67f2f26b8eb180796b8f8b0681ec9354"
server_key_version: "encryptionkey@0"
I20260924 04:03:59.388315 10474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:03:59.393052 10474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:03:59.393667 10474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:03:59.393860 10474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:03:59.394454 10474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:03:59.395378 10474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:03:59.395428 10474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.395470 10474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:03:59.395494 10474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:03:59.404503 10474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46695
I20260924 04:03:59.404549 10601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46695 every 8 connection(s)
I20260924 04:03:59.405457 10474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:03:59.407465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 10474
I20260924 04:03:59.407577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:03:59.407835 392 external_mini_cluster.cc:1428] Setting key 52f7d1d24a2a609484df0671691429cd
I20260924 04:03:59.411787 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:59.407345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38355 (local address 127.0.98.62:36599)
0924 04:03:59.407511 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:59.407516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:59.408814 (+ 1298us) server_negotiation.cc:407] Connection header received
0924 04:03:59.410309 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:59.410312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:59.410347 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:59.410421 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:59.411461 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:03:59.411464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:03:59.411466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:03:59.411477 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:03:59.411496 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:03:59.411504 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:03:59.411566 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:03:59.411655 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:03:59.411687 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:03:59.412659 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:03:59.407829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36599 (local address 127.0.98.3:38355)
0924 04:03:59.408658 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 04:03:59.408714 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:03:59.410122 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:03:59.410577 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:03:59.410583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:03:59.410673 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:03:59.411217 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:03:59.411219 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:03:59.411260 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:03:59.411328 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:03:59.411563 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:03:59.411567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:03:59.411570 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:03:59.411687 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:03:59.411880 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":125,"threads_started":1}
I20260924 04:03:59.413659 10602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36599
I20260924 04:03:59.413880 10602 heartbeater.cc:464] Registering TS with master...
I20260924 04:03:59.414279 10602 heartbeater.cc:511] Master 127.0.98.62:36599 requested a full tablet report, sending...
I20260924 04:03:59.415385 10152 ts_manager.cc:194] Registered new tserver with Master: fa4a9e2b74d540569921a4424de4a941 (127.0.98.3:46695)
I20260924 04:03:59.421777 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:03:59.424363 10191 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:03:59.423606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.0.98.62:36599)
0924 04:03:59.423721 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:03:59.423726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:03:59.423860 (+ 134us) server_negotiation.cc:407] Connection header received
0924 04:03:59.424001 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:03:59.424003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:03:59.424036 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:03:59.424097 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:03:59.424263 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37106: BlockingRecv error: recv got EOF from 127.0.0.1:37106 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:03:59.424553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 10212
I20260924 04:03:59.430459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 10343
I20260924 04:03:59.435135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 10474
I20260924 04:03:59.439723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 10121
2026-09-24T04:03:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (791 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:03:59 dist-test-slave-zcsk krb5kdc[10612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:03:59 dist-test-slave-zcsk krb5kdc[10612](info): set up 2 sockets
Sep 24 04:03:59 dist-test-slave-zcsk krb5kdc[10612](info): commencing operation
krb5kdc: starting...
W20260924 04:04:00.396709 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:04:00 dist-test-slave-zcsk krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:04:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:04:00.515165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:04:00.623643 10625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:04:00.623961 10625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:04:00.624009 10625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:04:00.628077 10625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:04:00.628147 10625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:04:00.628166 10625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:04:00.628182 10625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:04:00.628196 10625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:04:00.633625 10625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:04:00.634743 10625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:04:00.635754 10625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:04:00.644306 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:00.644338 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:00.644383 10625 server_base.cc:1034] running on GCE node
W20260924 04:04:00.644310 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:00.645154 10625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:04:00.646329 10625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:04:00.647500 10625 hybrid_clock.cc:648] HybridClock initialized: now 1790222640647482 us; error 33 us; skew 500 ppm
Sep 24 04:04:00 dist-test-slave-zcsk krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:04:00.650303 10625 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:04:00.651302 10625 webserver.cc:460] Webserver started at http://127.0.98.62:34585/ using document root <none> and password file <none>
I20260924 04:04:00.651888 10625 fs_manager.cc:359] Metadata directory not provided
I20260924 04:04:00.651997 10625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:04:00.652206 10625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:04:00.654260 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "5c6c048cad6042feb72fbdd656351bfd"
format_stamp: "Formatted at 2026-09-24 04:04:00 on dist-test-slave-zcsk"
server_key: "ae973bc27d5afe38bd9985aa7bc9fb1d"
server_key_iv: "9742550414592b67a032c74c5bee8349"
server_key_version: "encryptionkey@0"
I20260924 04:04:00.654801 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c6c048cad6042feb72fbdd656351bfd"
format_stamp: "Formatted at 2026-09-24 04:04:00 on dist-test-slave-zcsk"
server_key: "ae973bc27d5afe38bd9985aa7bc9fb1d"
server_key_iv: "9742550414592b67a032c74c5bee8349"
server_key_version: "encryptionkey@0"
I20260924 04:04:00.658717 10625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 04:04:00.661350 10640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:00.662683 10625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:04:00.662838 10625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "5c6c048cad6042feb72fbdd656351bfd"
format_stamp: "Formatted at 2026-09-24 04:04:00 on dist-test-slave-zcsk"
server_key: "ae973bc27d5afe38bd9985aa7bc9fb1d"
server_key_iv: "9742550414592b67a032c74c5bee8349"
server_key_version: "encryptionkey@0"
I20260924 04:04:00.662950 10625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:04:00.681602 10625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:04:00.684983 10625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:04:00.685169 10625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:04:00.691817 10625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38509
I20260924 04:04:00.692086 10692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38509 every 8 connection(s)
I20260924 04:04:00.692824 10625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:04:00.695739 10693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:04:00.701139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 10625
I20260924 04:04:00.701278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:04:00.701570 392 external_mini_cluster.cc:1428] Setting key 84bd11e85770d41297b3af8051e3d137
I20260924 04:04:00.701901 10693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd: Bootstrap starting.
I20260924 04:04:00.704615 10693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd: Neither blocks nor log segments found. Creating new log.
I20260924 04:04:00.705554 10693 log.cc:824] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:04:00.707707 10693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd: No bootstrap required, opened a new log
Sep 24 04:04:00 dist-test-slave-zcsk krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:04:00.712275 10693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c048cad6042feb72fbdd656351bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260924 04:04:00.712580 10693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:04:00.712646 10693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6c048cad6042feb72fbdd656351bfd, State: Initialized, Role: FOLLOWER
I20260924 04:04:00.713158 10693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c048cad6042feb72fbdd656351bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260924 04:04:00.713299 10693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:04:00.713368 10693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:04:00.713474 10693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:04:00.714612 10693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c048cad6042feb72fbdd656351bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260924 04:04:00.715000 10693 leader_election.cc:302] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c6c048cad6042feb72fbdd656351bfd; no voters:
I20260924 04:04:00.715346 10693 leader_election.cc:288] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:04:00.715523 10698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:04:00.715801 10698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [term 1 LEADER]: Becoming Leader. State: Replica: 5c6c048cad6042feb72fbdd656351bfd, State: Running, Role: LEADER
I20260924 04:04:00.716214 10698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c048cad6042feb72fbdd656351bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260924 04:04:00.716720 10693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:04:00.717322 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:00.703091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.0.98.62:38509)
0924 04:04:00.703549 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:00.703571 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:00.703607 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:04:00.704315 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:00.704337 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:00.704693 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:00.705010 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:00.705976 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:00.706983 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:00.708157 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:00.708667 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:00.708799 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:00.711782 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:00.711788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:00.711803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:00.711833 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:00.713773 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:00.714262 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:00.714266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:00.714271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:00.714398 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:00.714737 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:00.714741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:00.714742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:00.715238 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:00.715442 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:00.715778 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 04:04:00.716294 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":157,"threads_started":1}
I20260924 04:04:00.717517 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c6c048cad6042feb72fbdd656351bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c048cad6042feb72fbdd656351bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } } }
I20260924 04:04:00.717623 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [sys.catalog]: This master's current role is: LEADER
I20260924 04:04:00.717741 10699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c6c048cad6042feb72fbdd656351bfd. Latest consensus state: current_term: 1 leader_uuid: "5c6c048cad6042feb72fbdd656351bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c048cad6042feb72fbdd656351bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } } }
I20260924 04:04:00.717815 10699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd [sys.catalog]: This master's current role is: LEADER
I20260924 04:04:00.718021 10707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:04:00.721805 10707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:04:00.727562 10707 catalog_manager.cc:1347] Generated new cluster ID: 8bbbd5a45bea4beb914ad624017f3fea
I20260924 04:04:00.727646 10707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:04:00.760462 10707 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:04:00.761782 10707 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:04:00.774349 10707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd: Generated new TSK 0
I20260924 04:04:00.775092 10707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:04:00.806631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:04:00.927392 10719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:04:00.927657 10719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:04:00.927699 10719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:04:00.931696 10719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:04:00.931757 10719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:04:00.931818 10719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:04:00.937214 10719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:04:00.938246 10719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:04:00.939260 10719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:04:00.949142 10727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:00.949142 10725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:00.949180 10724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:00.949411 10719 server_base.cc:1034] running on GCE node
I20260924 04:04:00.949826 10719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:04:00.950510 10719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:04:00.951673 10719 hybrid_clock.cc:648] HybridClock initialized: now 1790222640951651 us; error 34 us; skew 500 ppm
Sep 24 04:04:00 dist-test-slave-zcsk krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:04:00.959237 10719 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:04:00.960287 10719 webserver.cc:460] Webserver started at http://127.0.98.1:36379/ using document root <none> and password file <none>
I20260924 04:04:00.960844 10719 fs_manager.cc:359] Metadata directory not provided
I20260924 04:04:00.960911 10719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:04:00.961109 10719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:04:00.963246 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "8e947d630ba34031ab2de190b0ff6cfd"
format_stamp: "Formatted at 2026-09-24 04:04:00 on dist-test-slave-zcsk"
server_key: "07bb95afcad0692f3ba9f784163eb1f5"
server_key_iv: "5b5a6549fdf7aad39a06a3580c9004c8"
server_key_version: "encryptionkey@0"
I20260924 04:04:00.963794 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8e947d630ba34031ab2de190b0ff6cfd"
format_stamp: "Formatted at 2026-09-24 04:04:00 on dist-test-slave-zcsk"
server_key: "07bb95afcad0692f3ba9f784163eb1f5"
server_key_iv: "5b5a6549fdf7aad39a06a3580c9004c8"
server_key_version: "encryptionkey@0"
I20260924 04:04:00.967713 10719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:04:00.970323 10734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:00.971686 10719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:04:00.971810 10719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "8e947d630ba34031ab2de190b0ff6cfd"
format_stamp: "Formatted at 2026-09-24 04:04:00 on dist-test-slave-zcsk"
server_key: "07bb95afcad0692f3ba9f784163eb1f5"
server_key_iv: "5b5a6549fdf7aad39a06a3580c9004c8"
server_key_version: "encryptionkey@0"
I20260924 04:04:00.971962 10719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:04:01.003577 10719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:04:01.006834 10719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:04:01.007038 10719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:04:01.007647 10719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:04:01.008514 10719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:04:01.008563 10719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:01.008618 10719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:04:01.008646 10719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:01.017176 10719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40093
I20260924 04:04:01.017241 10847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40093 every 8 connection(s)
I20260924 04:04:01.018133 10719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:04:01.023821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 10719
I20260924 04:04:01.023952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:04:01.024266 392 external_mini_cluster.cc:1428] Setting key 2d91bf85e0fa43051183ddae3c149bdf
Sep 24 04:04:01 dist-test-slave-zcsk krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:04:01.032428 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.019949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47609 (local address 127.0.98.62:38509)
0924 04:04:01.020081 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.020085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.021107 (+ 1022us) server_negotiation.cc:407] Connection header received
0924 04:04:01.022037 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.022040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.022100 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.022173 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:01.023379 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.023942 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.025369 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.025670 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.025760 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.028743 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:01.028746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:01.028748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:01.028774 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:01.030172 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:01.030811 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:01.030814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:01.030816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:01.030897 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:01.031263 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:01.031265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:01.031267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:01.031437 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:01.031507 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.032192 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.032302 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:04:01.033658 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.020320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:47609)
0924 04:04:01.020953 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.020994 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:01.021819 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:01.022364 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:01.022371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:01.022735 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:04:01.023233 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.023246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.024080 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.024084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.025257 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.025267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.025841 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.025846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.026051 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.026621 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:04:01.026642 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:04:01.028544 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:04:01.030302 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:01.030309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:01.030330 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:01.030697 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:01.030979 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:01.030983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:01.030984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:01.031182 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:01.031529 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:04:01.031536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:04:01.031894 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 04:04:01.032160 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 04:04:01.032541 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:04:01.034950 10848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38509
I20260924 04:04:01.035171 10848 heartbeater.cc:464] Registering TS with master...
I20260924 04:04:01.035642 10848 heartbeater.cc:511] Master 127.0.98.62:38509 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:04:01.037377 10657 ts_manager.cc:194] Registered new tserver with Master: 8e947d630ba34031ab2de190b0ff6cfd (127.0.98.1:40093)
I20260924 04:04:01.038614 10657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:04:01.050899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:04:01.161113 10853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:04:01.161435 10853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:04:01.161484 10853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:04:01.166043 10853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:04:01.166126 10853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:04:01.166193 10853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:04:01.172223 10853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:04:01.173306 10853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:04:01.174332 10853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:04:01.184465 10858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:01.184456 10859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:01.184435 10861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:01.184698 10853 server_base.cc:1034] running on GCE node
I20260924 04:04:01.185225 10853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:04:01.185875 10853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:04:01.187065 10853 hybrid_clock.cc:648] HybridClock initialized: now 1790222641187032 us; error 41 us; skew 500 ppm
Sep 24 04:04:01 dist-test-slave-zcsk krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:04:01.189885 10853 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:04:01.190775 10853 webserver.cc:460] Webserver started at http://127.0.98.2:44353/ using document root <none> and password file <none>
I20260924 04:04:01.191319 10853 fs_manager.cc:359] Metadata directory not provided
I20260924 04:04:01.191385 10853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:04:01.191593 10853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:04:01.193660 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc"
format_stamp: "Formatted at 2026-09-24 04:04:01 on dist-test-slave-zcsk"
server_key: "4de8a0274ba3548c639c9a90d3344e2d"
server_key_iv: "25c13967ef9446782d19ce2d0f58ecf4"
server_key_version: "encryptionkey@0"
I20260924 04:04:01.194195 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc"
format_stamp: "Formatted at 2026-09-24 04:04:01 on dist-test-slave-zcsk"
server_key: "4de8a0274ba3548c639c9a90d3344e2d"
server_key_iv: "25c13967ef9446782d19ce2d0f58ecf4"
server_key_version: "encryptionkey@0"
I20260924 04:04:01.198042 10853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:04:01.200627 10868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:01.201957 10853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:04:01.202076 10853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc"
format_stamp: "Formatted at 2026-09-24 04:04:01 on dist-test-slave-zcsk"
server_key: "4de8a0274ba3548c639c9a90d3344e2d"
server_key_iv: "25c13967ef9446782d19ce2d0f58ecf4"
server_key_version: "encryptionkey@0"
I20260924 04:04:01.202180 10853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:04:01.226198 10853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:04:01.229797 10853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:04:01.230015 10853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:04:01.230674 10853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:04:01.231580 10853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:04:01.231635 10853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:01.231683 10853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:04:01.231715 10853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:01.241190 10853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43685
I20260924 04:04:01.241240 10981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43685 every 8 connection(s)
I20260924 04:04:01.242252 10853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:04:01.247200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 10853
I20260924 04:04:01.247303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:04:01.247547 392 external_mini_cluster.cc:1428] Setting key 67c28a0d61897ea649b6b0baf91e6407
Sep 24 04:04:01 dist-test-slave-zcsk krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:04:01.256811 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.244283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40715 (local address 127.0.98.62:38509)
0924 04:04:01.244409 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.244413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.245528 (+ 1115us) server_negotiation.cc:407] Connection header received
0924 04:04:01.246431 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.246434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.246491 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.246564 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:01.247865 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.248426 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.249564 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.249808 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.249895 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.252890 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:01.252894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:01.252898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:01.252928 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:01.254468 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:01.255131 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:01.255134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:01.255135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:01.255211 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:01.255593 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:01.255595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:01.255596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:01.255779 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:01.255848 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.256573 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.256685 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:04:01.257880 10984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.244644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:40715)
0924 04:04:01.245369 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.245412 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:01.246222 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:01.246754 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:01.246761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:01.247160 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:04:01.247727 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.247742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.248550 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.248553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.249444 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.249453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.249951 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.249955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.250128 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.250869 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:04:01.250890 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:04:01.252647 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:04:01.254600 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:01.254608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:01.254628 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:01.255033 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:01.255305 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:01.255308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:01.255310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:01.255510 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:01.255882 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:04:01.255889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:04:01.256284 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 04:04:01.256544 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 04:04:01.256929 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":117,"threads_started":1}
I20260924 04:04:01.259014 10982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:04:01.259310 10982 heartbeater.cc:464] Registering TS with master...
I20260924 04:04:01.259929 10982 heartbeater.cc:511] Master 127.0.98.62:38509 requested a full tablet report, sending...
I20260924 04:04:01.261142 10657 ts_manager.cc:194] Registered new tserver with Master: bcbd35d280ad4bfbbe7bf7c7b28a9fcc (127.0.98.2:43685)
I20260924 04:04:01.261924 10657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:04:01.276415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:04:01.386724 10987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:04:01.387013 10987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:04:01.387058 10987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:04:01.391059 10987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:04:01.391131 10987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:04:01.391191 10987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:04:01.396625 10987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:04:01.397665 10987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:04:01.398655 10987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:04:01.408152 10993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:01.408191 10987 server_base.cc:1034] running on GCE node
W20260924 04:04:01.408366 10995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:01.408217 10992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:01.408738 10987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:04:01.409370 10987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:04:01.410542 10987 hybrid_clock.cc:648] HybridClock initialized: now 1790222641410523 us; error 37 us; skew 500 ppm
Sep 24 04:04:01 dist-test-slave-zcsk krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:04:01.413232 10987 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:04:01.414099 10987 webserver.cc:460] Webserver started at http://127.0.98.3:35953/ using document root <none> and password file <none>
I20260924 04:04:01.414628 10987 fs_manager.cc:359] Metadata directory not provided
I20260924 04:04:01.414695 10987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:04:01.414932 10987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:04:01.416982 10987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "d794526384944b129b2d8a1162561978"
format_stamp: "Formatted at 2026-09-24 04:04:01 on dist-test-slave-zcsk"
server_key: "fa219cc181c40718c7a7e6cd04ef356b"
server_key_iv: "075eec76b241b722b8c54b82b5302182"
server_key_version: "encryptionkey@0"
I20260924 04:04:01.417517 10987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d794526384944b129b2d8a1162561978"
format_stamp: "Formatted at 2026-09-24 04:04:01 on dist-test-slave-zcsk"
server_key: "fa219cc181c40718c7a7e6cd04ef356b"
server_key_iv: "075eec76b241b722b8c54b82b5302182"
server_key_version: "encryptionkey@0"
I20260924 04:04:01.421154 10987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:04:01.423453 11002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:01.424671 10987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:04:01.424803 10987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "d794526384944b129b2d8a1162561978"
format_stamp: "Formatted at 2026-09-24 04:04:01 on dist-test-slave-zcsk"
server_key: "fa219cc181c40718c7a7e6cd04ef356b"
server_key_iv: "075eec76b241b722b8c54b82b5302182"
server_key_version: "encryptionkey@0"
I20260924 04:04:01.424896 10987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:04:01.460901 10987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:04:01.464457 10987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:04:01.464649 10987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:04:01.465262 10987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:04:01.466152 10987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:04:01.466207 10987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:01.466264 10987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:04:01.466289 10987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:01.474951 10987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44627
I20260924 04:04:01.475001 11115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44627 every 8 connection(s)
I20260924 04:04:01.475975 10987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:04:01.483086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 10987
I20260924 04:04:01.483208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:04:01.483527 392 external_mini_cluster.cc:1428] Setting key d00bb6ebabee2d32ed8dcce72ec51f41
Sep 24 04:04:01 dist-test-slave-zcsk krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:04:01.489480 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.477918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51459 (local address 127.0.98.62:38509)
0924 04:04:01.478051 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.478055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.479130 (+ 1075us) server_negotiation.cc:407] Connection header received
0924 04:04:01.480010 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.480013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.480061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.480136 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:01.481449 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.482011 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.483262 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.483569 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.483676 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.486260 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:01.486264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:01.486266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:01.486289 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:01.487539 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:01.488032 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:01.488036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:01.488038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:01.488113 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:01.488435 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:01.488437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:01.488439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:01.488609 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:01.488691 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.489194 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.489324 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:04:01.490268 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.478312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:51459)
0924 04:04:01.478956 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.478998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:01.479798 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:01.480316 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:01.480324 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:01.480711 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:04:01.481299 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.481320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.482123 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.482138 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.483074 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.483083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.483726 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.483730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.483891 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.484595 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:04:01.484613 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:04:01.486111 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:04:01.487645 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:01.487655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:01.487671 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:01.487943 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:01.488191 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:01.488194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:01.488196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:01.488342 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:01.488704 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:04:01.488710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:04:01.488955 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 04:04:01.489121 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:04:01.489460 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":114,"threads_started":1}
I20260924 04:04:01.491348 11116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38509
I20260924 04:04:01.491572 11116 heartbeater.cc:464] Registering TS with master...
I20260924 04:04:01.492087 11116 heartbeater.cc:511] Master 127.0.98.62:38509 requested a full tablet report, sending...
I20260924 04:04:01.493076 10657 ts_manager.cc:194] Registered new tserver with Master: d794526384944b129b2d8a1162561978 (127.0.98.3:44627)
I20260924 04:04:01.493634 10657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51459
I20260924 04:04:01.497238 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:04:01.506675 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.499297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.0.98.62:38509)
0924 04:04:01.499413 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.499416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.499577 (+ 161us) server_negotiation.cc:407] Connection header received
0924 04:04:01.499688 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.499690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.499729 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.499801 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:01.500553 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.501077 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.501987 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.502240 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.502341 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.503749 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:01.503752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:01.503755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:01.503778 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:01.505176 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:01.505584 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:01.505587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:01.505588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:01.505662 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:01.505865 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:01.505867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:01.505868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:01.506104 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:01.506197 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.506461 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.506555 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:04:01.509640 10657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:04:01.512635 10657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:04:01.526970 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.521899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.0.98.3:44627)
0924 04:04:01.522427 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.522438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.522461 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:04:01.522519 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.522523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.522741 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.522896 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:01.523170 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.524155 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.525580 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.526314 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.526406 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.526553 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.526725 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.526817 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":83,"threads_started":1}
I20260924 04:04:01.526960 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.521246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44627 (local address 127.0.0.1:42966)
0924 04:04:01.521800 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.521852 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:01.521987 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:01.522881 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:01.522887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:01.522907 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:01.523049 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.523057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.524267 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.524270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.525449 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.525459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.526425 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.526429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.526537 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.526586 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 04:04:01.526688 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:04:01.526814 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"spinlock_wait_cycles":3968,"thread_start_us":96,"threads_started":1}
I20260924 04:04:01.527307 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.521081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40093 (local address 127.0.0.1:45292)
0924 04:04:01.521799 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.521852 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:01.521988 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:01.523551 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:01.523553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:01.523571 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:01.523651 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.523656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.524916 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.524920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.525894 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.525902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.526931 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.526935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.527017 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.527038 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:04:01.527086 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:04:01.527151 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":2,"spinlock_wait_cycles":11520,"thread_start_us":113,"threads_started":1}
I20260924 04:04:01.527436 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.521501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43685 (local address 127.0.0.1:49666)
0924 04:04:01.522329 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.522339 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:01.522410 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:01.523147 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:01.523149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:01.523167 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:01.523268 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.523273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.524692 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.524695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.525994 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.526002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.527114 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.527118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.527192 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.527213 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:04:01.527272 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 04:04:01.527329 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":178,"thread_start_us":98,"threads_started":1}
I20260924 04:04:01.527833 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.521910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.0.98.2:43685)
0924 04:04:01.522614 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.522623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.522640 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:04:01.522696 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.522699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.522871 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.523001 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:01.523382 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.524556 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.526087 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.526829 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.526929 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.527061 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.527589 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.527685 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":155,"threads_started":1}
I20260924 04:04:01.527988 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.521322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.0.98.1:40093)
0924 04:04:01.523178 (+ 1856us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.523201 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.523221 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:04:01.523274 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.523277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.523428 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.523565 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:01.523739 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.524803 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.526004 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.526795 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.527574 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.527712 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.527757 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.527834 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1759,"thread_start_us":94,"threads_started":1}
I20260924 04:04:01.530431 10782 tablet_service.cc:1467] Processing CreateTablet for tablet 694bb42912ff4cd3b78491ca9b185d4f (DEFAULT_TABLE table=test-table [id=07ac4429817d406dacccdd01a355b3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:04:01.530503 11050 tablet_service.cc:1467] Processing CreateTablet for tablet 694bb42912ff4cd3b78491ca9b185d4f (DEFAULT_TABLE table=test-table [id=07ac4429817d406dacccdd01a355b3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:04:01.530504 10916 tablet_service.cc:1467] Processing CreateTablet for tablet 694bb42912ff4cd3b78491ca9b185d4f (DEFAULT_TABLE table=test-table [id=07ac4429817d406dacccdd01a355b3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:04:01.531306 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694bb42912ff4cd3b78491ca9b185d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:04:01.531466 10916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694bb42912ff4cd3b78491ca9b185d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:04:01.531474 11050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 694bb42912ff4cd3b78491ca9b185d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:04:01.537917 11133 tablet_bootstrap.cc:492] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd: Bootstrap starting.
I20260924 04:04:01.538686 11135 tablet_bootstrap.cc:492] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: Bootstrap starting.
I20260924 04:04:01.539004 11134 tablet_bootstrap.cc:492] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: Bootstrap starting.
I20260924 04:04:01.540419 11133 tablet_bootstrap.cc:654] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd: Neither blocks nor log segments found. Creating new log.
I20260924 04:04:01.540855 11135 tablet_bootstrap.cc:654] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: Neither blocks nor log segments found. Creating new log.
I20260924 04:04:01.541149 11133 log.cc:824] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:04:01.541280 11134 tablet_bootstrap.cc:654] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: Neither blocks nor log segments found. Creating new log.
I20260924 04:04:01.542026 11135 log.cc:824] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: Log is configured to *not* fsync() on all Append() calls
I20260924 04:04:01.542026 11134 log.cc:824] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:04:01.543054 11133 tablet_bootstrap.cc:492] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd: No bootstrap required, opened a new log
I20260924 04:04:01.543246 11133 ts_tablet_manager.cc:1397] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:04:01.543781 11134 tablet_bootstrap.cc:492] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: No bootstrap required, opened a new log
I20260924 04:04:01.544056 11134 ts_tablet_manager.cc:1397] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:04:01.544247 11135 tablet_bootstrap.cc:492] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: No bootstrap required, opened a new log
I20260924 04:04:01.544461 11135 ts_tablet_manager.cc:1397] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:04:01.547079 11133 raft_consensus.cc:348] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.547313 11133 raft_consensus.cc:374] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:04:01.547390 11133 raft_consensus.cc:729] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e947d630ba34031ab2de190b0ff6cfd, State: Initialized, Role: FOLLOWER
I20260924 04:04:01.547816 11133 consensus_queue.cc:260] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.547833 11134 raft_consensus.cc:348] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.548120 11134 raft_consensus.cc:374] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:04:01.548244 11134 raft_consensus.cc:729] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbd35d280ad4bfbbe7bf7c7b28a9fcc, State: Initialized, Role: FOLLOWER
I20260924 04:04:01.548178 11135 raft_consensus.cc:348] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.548439 11135 raft_consensus.cc:374] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:04:01.548472 10848 heartbeater.cc:503] Master 127.0.98.62:38509 was elected leader, sending a full tablet report...
I20260924 04:04:01.548492 11135 raft_consensus.cc:729] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d794526384944b129b2d8a1162561978, State: Initialized, Role: FOLLOWER
I20260924 04:04:01.548720 11134 consensus_queue.cc:260] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.548840 11135 consensus_queue.cc:260] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.549490 10982 heartbeater.cc:503] Master 127.0.98.62:38509 was elected leader, sending a full tablet report...
I20260924 04:04:01.549705 11134 ts_tablet_manager.cc:1428] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:04:01.549712 11135 ts_tablet_manager.cc:1428] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:04:01.549976 11116 heartbeater.cc:503] Master 127.0.98.62:38509 was elected leader, sending a full tablet report...
I20260924 04:04:01.552241 11133 ts_tablet_manager.cc:1428] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260924 04:04:01.627279 11141 raft_consensus.cc:482] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:04:01.627498 11141 raft_consensus.cc:504] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.628808 11141 leader_election.cc:288] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e947d630ba34031ab2de190b0ff6cfd (127.0.98.1:40093), bcbd35d280ad4bfbbe7bf7c7b28a9fcc (127.0.98.2:43685)
I20260924 04:04:01.632951 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.629131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43685 (local address 127.0.98.3:48167)
0924 04:04:01.629264 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.629281 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:01.629403 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:01.629746 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:01.629749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:01.629771 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:01.629893 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.629900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.630800 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.630804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.631690 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.631699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.632501 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.632506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.632619 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.632648 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:04:01.632728 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:04:01.632805 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260924 04:04:01.632967 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.629215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48167 (local address 127.0.98.2:43685)
0924 04:04:01.629347 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.629352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.629371 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:04:01.629536 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.629540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.629595 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.629697 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:01.630029 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.630668 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.631831 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.632359 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.632442 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.632500 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.632751 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.632842 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:04:01.633162 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.629131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40093 (local address 127.0.98.3:45257)
0924 04:04:01.629642 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.629654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:01.629745 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:01.630016 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:01.630019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:01.630040 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:01.630161 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.630169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.631026 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.631029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.631936 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:01.631946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.632756 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.632760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:01.632874 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.632897 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:04:01.632969 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:04:01.633032 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":159,"thread_start_us":157,"threads_started":1}
I20260924 04:04:01.633214 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.629288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45257 (local address 127.0.98.1:40093)
0924 04:04:01.629472 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.629476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.629752 (+ 276us) server_negotiation.cc:407] Connection header received
0924 04:04:01.629816 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.629819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.629882 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.629969 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:01.630312 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.630901 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.632099 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.632612 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.632735 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.632811 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.633003 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.633086 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260924 04:04:01.633845 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694bb42912ff4cd3b78491ca9b185d4f" candidate_uuid: "d794526384944b129b2d8a1162561978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e947d630ba34031ab2de190b0ff6cfd" is_pre_election: true
I20260924 04:04:01.634027 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694bb42912ff4cd3b78491ca9b185d4f" candidate_uuid: "d794526384944b129b2d8a1162561978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" is_pre_election: true
I20260924 04:04:01.634219 10802 raft_consensus.cc:2454] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d794526384944b129b2d8a1162561978 in term 0.
I20260924 04:04:01.634290 10936 raft_consensus.cc:2454] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d794526384944b129b2d8a1162561978 in term 0.
I20260924 04:04:01.634778 11006 leader_election.cc:302] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e947d630ba34031ab2de190b0ff6cfd, d794526384944b129b2d8a1162561978; no voters:
I20260924 04:04:01.635073 11141 raft_consensus.cc:2781] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:04:01.635159 11141 raft_consensus.cc:482] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:04:01.635210 11141 raft_consensus.cc:3037] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:04:01.636264 11141 raft_consensus.cc:504] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.636613 11141 leader_election.cc:288] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [CANDIDATE]: Term 1 election: Requested vote from peers 8e947d630ba34031ab2de190b0ff6cfd (127.0.98.1:40093), bcbd35d280ad4bfbbe7bf7c7b28a9fcc (127.0.98.2:43685)
I20260924 04:04:01.636961 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694bb42912ff4cd3b78491ca9b185d4f" candidate_uuid: "d794526384944b129b2d8a1162561978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc"
I20260924 04:04:01.637001 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "694bb42912ff4cd3b78491ca9b185d4f" candidate_uuid: "d794526384944b129b2d8a1162561978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e947d630ba34031ab2de190b0ff6cfd"
I20260924 04:04:01.637080 10936 raft_consensus.cc:3037] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:04:01.637113 10802 raft_consensus.cc:3037] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:04:01.638100 10936 raft_consensus.cc:2454] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d794526384944b129b2d8a1162561978 in term 1.
I20260924 04:04:01.638113 10802 raft_consensus.cc:2454] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d794526384944b129b2d8a1162561978 in term 1.
I20260924 04:04:01.638370 11006 leader_election.cc:302] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8e947d630ba34031ab2de190b0ff6cfd, bcbd35d280ad4bfbbe7bf7c7b28a9fcc, d794526384944b129b2d8a1162561978; no voters:
I20260924 04:04:01.638578 11141 raft_consensus.cc:2781] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:04:01.638823 11141 raft_consensus.cc:686] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 1 LEADER]: Becoming Leader. State: Replica: d794526384944b129b2d8a1162561978, State: Running, Role: LEADER
I20260924 04:04:01.639160 11141 consensus_queue.cc:237] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } }
I20260924 04:04:01.642861 10655 catalog_manager.cc:5495] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 reported cstate change: term changed from 0 to 1, leader changed from <none> to d794526384944b129b2d8a1162561978 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d794526384944b129b2d8a1162561978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d794526384944b129b2d8a1162561978" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 } health_report { overall_health: UNKNOWN } } }
I20260924 04:04:01.681104 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.677403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.0.98.3:44627)
0924 04:04:01.677529 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:01.677532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:01.677580 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:04:01.677656 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:01.677658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:01.677723 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:01.677806 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:04:01.678155 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.678708 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.679595 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:01.679822 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:01.679934 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:01.680282 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:04:01.680752 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:04:01.680860 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:01.680917 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:04:01.680972 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:04:01.687160 10936 raft_consensus.cc:1261] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 1 FOLLOWER]: Refusing update from remote peer d794526384944b129b2d8a1162561978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:04:01.687160 10802 raft_consensus.cc:1261] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 1 FOLLOWER]: Refusing update from remote peer d794526384944b129b2d8a1162561978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:04:01.687877 11141 consensus_queue.cc:1035] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcbd35d280ad4bfbbe7bf7c7b28a9fcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:04:01.688143 11143 consensus_queue.cc:1035] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e947d630ba34031ab2de190b0ff6cfd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:04:01.697263 11146 mvcc.cc:205] Tried to move back new op lower bound from 7332751940342935552 to 7332751940156891136. Current Snapshot: MvccSnapshot[applied={T|T < 7332751940342935552}]
I20260924 04:04:01.699644 11147 mvcc.cc:205] Tried to move back new op lower bound from 7332751940342935552 to 7332751940156891136. Current Snapshot: MvccSnapshot[applied={T|T < 7332751940342935552}]
I20260924 04:04:01.708103 10655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:04:01.708290 10655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:04:01.710839 10655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c6c048cad6042feb72fbdd656351bfd: Sending DeleteTablet for 3 replicas of tablet 694bb42912ff4cd3b78491ca9b185d4f
I20260924 04:04:01.711644 10782 tablet_service.cc:1514] Processing DeleteTablet for tablet 694bb42912ff4cd3b78491ca9b185d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:04:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45292
I20260924 04:04:01.711644 11050 tablet_service.cc:1514] Processing DeleteTablet for tablet 694bb42912ff4cd3b78491ca9b185d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:04:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42966
I20260924 04:04:01.711941 10916 tablet_service.cc:1514] Processing DeleteTablet for tablet 694bb42912ff4cd3b78491ca9b185d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:04:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49666
I20260924 04:04:01.712165 11156 tablet_replica.cc:330] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd: stopping tablet replica
I20260924 04:04:01.712481 11157 tablet_replica.cc:330] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: stopping tablet replica
I20260924 04:04:01.712491 11156 raft_consensus.cc:2229] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:04:01.712848 11156 raft_consensus.cc:2258] T 694bb42912ff4cd3b78491ca9b185d4f P 8e947d630ba34031ab2de190b0ff6cfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:04:01.712836 11157 raft_consensus.cc:2229] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:04:01.713178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 10719
I20260924 04:04:01.713202 11155 tablet_replica.cc:330] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: stopping tablet replica
I20260924 04:04:01.714154 11155 raft_consensus.cc:2229] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:04:01.714380 11157 raft_consensus.cc:2258] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:04:01.714560 11155 raft_consensus.cc:2258] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:04:01.716270 11157 ts_tablet_manager.cc:1905] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:04:01.718035 11155 ts_tablet_manager.cc:1905] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:04:01.719300 11157 ts_tablet_manager.cc:1918] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:04:01.719465 11157 log.cc:1196] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/694bb42912ff4cd3b78491ca9b185d4f
I20260924 04:04:01.719830 11157 ts_tablet_manager.cc:1939] T 694bb42912ff4cd3b78491ca9b185d4f P bcbd35d280ad4bfbbe7bf7c7b28a9fcc: Deleting consensus metadata
I20260924 04:04:01.720723 11155 ts_tablet_manager.cc:1918] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 04:04:01.720781 10644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40093 (error 108)
I20260924 04:04:01.720876 11155 log.cc:1196] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/694bb42912ff4cd3b78491ca9b185d4f
I20260924 04:04:01.721207 11155 ts_tablet_manager.cc:1939] T 694bb42912ff4cd3b78491ca9b185d4f P d794526384944b129b2d8a1162561978: Deleting consensus metadata
I20260924 04:04:01.721184 10641 catalog_manager.cc:4841] TS bcbd35d280ad4bfbbe7bf7c7b28a9fcc (127.0.98.2:43685): tablet 694bb42912ff4cd3b78491ca9b185d4f (table test-table [id=07ac4429817d406dacccdd01a355b3de]) successfully deleted
I20260924 04:04:01.722000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 10853
I20260924 04:04:01.722361 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:01.722029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40093 (local address 127.0.0.1:45298)
0924 04:04:01.722203 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:01.722269 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260924 04:04:01.722790 10644 catalog_manager.cc:4584] TS 8e947d630ba34031ab2de190b0ff6cfd (127.0.98.1:40093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 694bb42912ff4cd3b78491ca9b185d4f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40093: connect: Connection refused (error 111)
I20260924 04:04:01.724232 10642 catalog_manager.cc:4841] TS d794526384944b129b2d8a1162561978 (127.0.98.3:44627): tablet 694bb42912ff4cd3b78491ca9b185d4f (table test-table [id=07ac4429817d406dacccdd01a355b3de]) successfully deleted
I20260924 04:04:01.727252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 10987
I20260924 04:04:01.732324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 10625
2026-09-24T04:04:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2294 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:04:01 dist-test-slave-zcsk krb5kdc[11161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:04:01 dist-test-slave-zcsk krb5kdc[11161](info): set up 2 sockets
Sep 24 04:04:01 dist-test-slave-zcsk krb5kdc[11161](info): commencing operation
krb5kdc: starting...
W20260924 04:04:02.688608 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:04:02 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:04:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:04:02.848523 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskeFjGAy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:04:05.278 UTC [11191] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:04:05.278 UTC [11191] LOG: listening on IPv4 address "127.0.98.20", port 36975
2026-09-24 04:04:05.281 UTC [11191] LOG: listening on Unix socket "/tmp/.s.PGSQL.36975"
2026-09-24 04:04:05.308 UTC [11194] LOG: database system was shut down at 2026-09-24 04:04:03 UTC
2026-09-24 04:04:05.311 UTC [11191] LOG: database system is ready to accept connections
I20260924 04:04:06.180465 392 mini_postgres.cc:96] Postgres bound to 36975
2026-09-24 04:04:06.186 UTC [11202] FATAL: database "slave" does not exist
127.0.98.20:36975 - accepting connections
I20260924 04:04:06.187364 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:04:06.201628 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:04:06.663151 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:04:06.663249 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-admin
2026-09-24 04:04:06,945 [I] DB FLAVOR :POSTGRES
2026-09-24 04:04:06,946 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:04:06,946 [I] Checking connection..
2026-09-24 04:04:06,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:04:07,195 [I] Checking connection passed.
2026-09-24 04:04:07,195 [I] --------- Verifying version history table ---------
2026-09-24 04:04:07,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:04:07,423 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:04:07,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:04:07,671 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:04:07,672 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:04:07,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:04:07.899 UTC [11266] WARNING: there is no transaction in progress
2026-09-24 04:04:07,904 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:04:07,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:04:08,153 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:04:08,153 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:04:08,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:04:08,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:04:08,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zcsk','N') ;"
2026-09-24 04:04:08,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:04:09,076 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:04:09,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:04:09,330 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:04:09,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:04:09,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:04:09,810 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:04:09,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:04:10.765 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.781 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.798 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.813 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.828 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.845 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.913 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.920 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.933 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.943 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.952 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.965 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.975 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.981 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.987 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:10.995 UTC [11374] WARNING: there is no transaction in progress
2026-09-24 04:04:11,606 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:04:11,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:04:11,854 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:04:11,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:04:12,110 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:04:12,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:04:12,352 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:04:12,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:04:12,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:04:12,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zcsk';"
2026-09-24 04:04:13,064 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:04:13,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:04:13,294 [I] DB_PATCHES have already been applied
I20260924 04:04:13.299387 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskeFjGAy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:04:13.299458 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:04:13.302641 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40139
Sep 24, 2026 4:04:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:04:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:04:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:04:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:04:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:04:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:04:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:04:14.307941 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.121s user 0.003s sys 0.007s
Sep 24, 2026 4:04:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:04:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:04:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:04:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:04:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:04:35.687456 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:04:35 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222675, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:04:35.788435 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:04:35.799098 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:04:36.726640 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:04:36.726739 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms
I20260924 04:04:36.726825 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:36975
2026-09-24 04:04:36,904 [I] DB FLAVOR :POSTGRES
2026-09-24 04:04:36,904 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:04:36,904 [I] Checking connection
2026-09-24 04:04:36,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:04:37,165 [I] connection success
2026-09-24 04:04:37,166 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:04:37,166 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:04:37,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:04:37,404 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:04:37,405 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:04:37,405 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:04:37,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:04:37,670 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:04:38.109416 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:04:38.296314 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:04:38.338893 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:04:38 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222675, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:04:38.706231 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:04:38.706399 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:04:38.706427 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:04:38.709975 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11554
I20260924 04:04:38.710139 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60821
Sep 24, 2026 4:04:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:04:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:04:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:04:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:04:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:04:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:04:40.766258 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.978s user 0.008s sys 0.154s
I20260924 04:04:40.766369 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:04:40.766480 392 mini_ranger_kms.cc:401] 127.0.98.20:60821/kms/v1/keys
Sep 24 04:04:40 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790222680, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:04:41 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222680, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:04:41 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222680, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:04:43 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:04:44.031505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35003
--ranger_config_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60821/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:04:44.143766 11605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:04:44.144119 11605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:04:44.144196 11605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:04:44.148249 11605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:04:44.148311 11605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:04:44.148337 11605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:04:44.148360 11605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:04:44.148387 11605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:04:44.153898 11605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60821/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35003
--ranger_config_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:04:44.154949 11605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:04:44.156010 11605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:04:44.164739 11610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:44.164985 11605 server_base.cc:1034] running on GCE node
W20260924 04:04:44.164813 11611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:44.165179 11613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:44.165905 11605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:04:44.167465 11605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:04:44.168659 11605 hybrid_clock.cc:648] HybridClock initialized: now 1790222684168633 us; error 41 us; skew 500 ppm
Sep 24 04:04:44 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:04:44.172070 11605 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:04:44.173048 11605 webserver.cc:460] Webserver started at http://127.0.98.62:34483/ using document root <none> and password file <none>
I20260924 04:04:44.173592 11605 fs_manager.cc:359] Metadata directory not provided
I20260924 04:04:44.173650 11605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:04:44.173940 11605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:04:44 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:04:44.222800 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "feb356f5659b48e09cf8c4f2ad54b847"
format_stamp: "Formatted at 2026-09-24 04:04:44 on dist-test-slave-zcsk"
server_key: "2cc9402aebcdd2d5ca48264d177edae5"
server_key_iv: "a656a90f35b2dcf9650f64f6d9595941"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:44.223527 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "feb356f5659b48e09cf8c4f2ad54b847"
format_stamp: "Formatted at 2026-09-24 04:04:44 on dist-test-slave-zcsk"
server_key: "2cc9402aebcdd2d5ca48264d177edae5"
server_key_iv: "a656a90f35b2dcf9650f64f6d9595941"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:44.228453 11605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 04:04:44.243657 11620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:44.245134 11605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:04:44.245261 11605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "feb356f5659b48e09cf8c4f2ad54b847"
format_stamp: "Formatted at 2026-09-24 04:04:44 on dist-test-slave-zcsk"
server_key: "2cc9402aebcdd2d5ca48264d177edae5"
server_key_iv: "a656a90f35b2dcf9650f64f6d9595941"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:44.245401 11605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:04:44.260156 11605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:04:44.263155 11605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:04:44.263342 11605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:04:44.271519 11605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35003
I20260924 04:04:44.271548 11672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35003 every 8 connection(s)
I20260924 04:04:44.272641 11605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:04:44.275492 11673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:04:44.279258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 11605
I20260924 04:04:44.279377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:04:44.279635 392 external_mini_cluster.cc:1428] Setting key 06e36a00c1e7f8ffe0620c673d54f0cf
I20260924 04:04:44.281910 11673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847: Bootstrap starting.
I20260924 04:04:44.285218 11673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847: Neither blocks nor log segments found. Creating new log.
I20260924 04:04:44.286115 11673 log.cc:824] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847: Log is configured to *not* fsync() on all Append() calls
I20260924 04:04:44.288335 11673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847: No bootstrap required, opened a new log
Sep 24 04:04:44 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:04:44.292313 11673 raft_consensus.cc:348] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb356f5659b48e09cf8c4f2ad54b847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } }
I20260924 04:04:44.292555 11673 raft_consensus.cc:374] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:04:44.292634 11673 raft_consensus.cc:729] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb356f5659b48e09cf8c4f2ad54b847, State: Initialized, Role: FOLLOWER
I20260924 04:04:44.293100 11673 consensus_queue.cc:260] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb356f5659b48e09cf8c4f2ad54b847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } }
I20260924 04:04:44.293234 11673 raft_consensus.cc:388] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:04:44.293324 11673 raft_consensus.cc:482] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:04:44.293427 11673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:04:44.294519 11673 raft_consensus.cc:504] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb356f5659b48e09cf8c4f2ad54b847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } }
I20260924 04:04:44.294967 11673 leader_election.cc:302] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feb356f5659b48e09cf8c4f2ad54b847; no voters:
I20260924 04:04:44.295356 11673 leader_election.cc:288] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:04:44.295451 11678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:04:44.295714 11678 raft_consensus.cc:686] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [term 1 LEADER]: Becoming Leader. State: Replica: feb356f5659b48e09cf8c4f2ad54b847, State: Running, Role: LEADER
I20260924 04:04:44.296113 11678 consensus_queue.cc:237] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb356f5659b48e09cf8c4f2ad54b847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } }
I20260924 04:04:44.296154 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:44.281205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.0.98.62:35003)
0924 04:04:44.281695 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:44.281715 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:44.281758 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:04:44.282618 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:44.282641 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:44.283102 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:44.283577 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:44.284362 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:44.285823 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:44.286997 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:44.287651 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:44.287830 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:44.290829 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:44.290835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:44.290853 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:44.290884 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:44.292847 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:44.293339 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:44.293345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:44.293351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:44.293474 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:44.293757 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:44.293760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:44.293762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:44.294282 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:44.294487 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:44.294759 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 04:04:44.295053 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":140,"threads_started":1}
I20260924 04:04:44.296782 11673 sys_catalog.cc:564] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:04:44.297340 11678 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feb356f5659b48e09cf8c4f2ad54b847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb356f5659b48e09cf8c4f2ad54b847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } } }
I20260924 04:04:44.297432 11678 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [sys.catalog]: This master's current role is: LEADER
I20260924 04:04:44.298882 11673 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:04:44.299407 11684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:04:44.301918 11679 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb356f5659b48e09cf8c4f2ad54b847. Latest consensus state: current_term: 1 leader_uuid: "feb356f5659b48e09cf8c4f2ad54b847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb356f5659b48e09cf8c4f2ad54b847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } } }
I20260924 04:04:44.302281 11679 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847 [sys.catalog]: This master's current role is: LEADER
I20260924 04:04:44.305590 11684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:04:44.312379 11684 catalog_manager.cc:1347] Generated new cluster ID: e7fa7ff01f2148de8af9271d8a8705d4
I20260924 04:04:44.312438 11684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:04:44.327718 11684 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:04:44.329216 11684 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:04:44.339211 11684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847: Generated new TSK 0
I20260924 04:04:44.340279 11684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:04:45.116096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:04:45.270196 11718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:04:45.270696 11718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:04:45.270814 11718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:04:45.277114 11718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:04:45.277201 11718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:04:45.277287 11718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:04:45.285094 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:04:45.286212 11718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:04:45.287277 11718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:04:45.297065 11728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:45.297065 11726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:45.297600 11725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:45.298231 11718 server_base.cc:1034] running on GCE node
I20260924 04:04:45.298710 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:04:45.299417 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:04:45.300595 11718 hybrid_clock.cc:648] HybridClock initialized: now 1790222685300565 us; error 43 us; skew 500 ppm
Sep 24 04:04:45 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:04:45.303333 11718 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:04:45.304519 11718 webserver.cc:460] Webserver started at http://127.0.98.1:44457/ using document root <none> and password file <none>
I20260924 04:04:45.305259 11718 fs_manager.cc:359] Metadata directory not provided
I20260924 04:04:45.305354 11718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:04:45.305591 11718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:04:45 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:04:45.321918 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "10b33d286b2644b7bec41d30f3322c74"
format_stamp: "Formatted at 2026-09-24 04:04:45 on dist-test-slave-zcsk"
server_key: "47674a9ff0f51a6ec8464826ed351fb0"
server_key_iv: "48105279abe401ed4e8faba27caa6cbe"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:45.322798 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "10b33d286b2644b7bec41d30f3322c74"
format_stamp: "Formatted at 2026-09-24 04:04:45 on dist-test-slave-zcsk"
server_key: "47674a9ff0f51a6ec8464826ed351fb0"
server_key_iv: "48105279abe401ed4e8faba27caa6cbe"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:45.329473 11718 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 04:04:45.360440 11735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:45.362322 11718 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260924 04:04:45.362497 11718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "10b33d286b2644b7bec41d30f3322c74"
format_stamp: "Formatted at 2026-09-24 04:04:45 on dist-test-slave-zcsk"
server_key: "47674a9ff0f51a6ec8464826ed351fb0"
server_key_iv: "48105279abe401ed4e8faba27caa6cbe"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:45.362658 11718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:04:45.406311 11718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:04:45.412887 11718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:04:45.413105 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:04:45.413866 11718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:04:45.414965 11718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:04:45.415030 11718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:45.415091 11718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:04:45.415118 11718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:45.426396 11718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36725
I20260924 04:04:45.426476 11850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36725 every 8 connection(s)
I20260924 04:04:45.427580 11718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:04:45.431293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 11718
I20260924 04:04:45.431418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:04:45.431768 392 external_mini_cluster.cc:1428] Setting key 6d4d60b5dadf3044e26c620cc71f359a
Sep 24 04:04:45 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:04:45.446170 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:45.429849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37999 (local address 127.0.98.62:35003)
0924 04:04:45.430119 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:45.430124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:45.432114 (+ 1990us) server_negotiation.cc:407] Connection header received
0924 04:04:45.433387 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:45.433390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:45.433444 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:45.433503 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:45.435246 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:45.436141 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:45.437915 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:45.438268 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:45.438349 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:45.441683 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:45.441689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:45.441693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:45.441730 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:45.443599 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:45.444276 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:45.444280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:45.444282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:45.444380 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:45.444795 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:45.444799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:45.444801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:45.445039 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:45.445136 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:45.445839 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 04:04:45.445981 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260924 04:04:45.448207 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:45.430366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35003 (local address 127.0.98.1:37999)
0924 04:04:45.430992 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:45.431994 (+ 1002us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:45.433129 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:45.433946 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:45.433957 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:45.434327 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:04:45.435057 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:45.435076 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:45.436299 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:45.436302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:45.437772 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:45.437783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:45.438566 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:45.438570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:45.438788 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:45.439474 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:04:45.439496 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:04:45.441386 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:04:45.443759 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:45.443766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:45.443785 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:45.444156 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:45.444495 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:45.444498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:45.444500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:45.444701 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:45.445159 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:04:45.445166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:04:45.445534 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 04:04:45.446628 (+ 1094us) client_negotiation.cc:240] Negotiation successful
0924 04:04:45.447018 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":138,"threads_started":1}
I20260924 04:04:45.450006 11851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35003
I20260924 04:04:45.450258 11851 heartbeater.cc:464] Registering TS with master...
I20260924 04:04:45.450862 11851 heartbeater.cc:511] Master 127.0.98.62:35003 requested a full tablet report, sending...
I20260924 04:04:45.452924 11637 ts_manager.cc:194] Registered new tserver with Master: 10b33d286b2644b7bec41d30f3322c74 (127.0.98.1:36725)
I20260924 04:04:45.454506 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:04:45.464742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:04:45 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:04:45.632786 11857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:04:45.633230 11857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:04:45.633319 11857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:04:45.639449 11857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:04:45.639539 11857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:04:45.639631 11857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:04:45.647698 11857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:04:45.649271 11857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:04:45.650735 11857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:04:45.666371 11866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:45.666563 11864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:45.666370 11863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:45.668128 11857 server_base.cc:1034] running on GCE node
I20260924 04:04:45.668676 11857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:04:45.669589 11857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:04:45.670781 11857 hybrid_clock.cc:648] HybridClock initialized: now 1790222685670757 us; error 41 us; skew 500 ppm
Sep 24 04:04:45 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:04:45.674439 11857 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:04:45.675689 11857 webserver.cc:460] Webserver started at http://127.0.98.2:40743/ using document root <none> and password file <none>
I20260924 04:04:45.676510 11857 fs_manager.cc:359] Metadata directory not provided
I20260924 04:04:45.676604 11857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:04:45.676839 11857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:04:45 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:04:45.697795 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "095c9523d1cc48a9baf3fbd40e3a190b"
format_stamp: "Formatted at 2026-09-24 04:04:45 on dist-test-slave-zcsk"
server_key: "33fc825dd2bfd76598f00ae619c5162b"
server_key_iv: "71231bf843efc924bdc99e3531f94d8c"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:45.698618 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "095c9523d1cc48a9baf3fbd40e3a190b"
format_stamp: "Formatted at 2026-09-24 04:04:45 on dist-test-slave-zcsk"
server_key: "33fc825dd2bfd76598f00ae619c5162b"
server_key_iv: "71231bf843efc924bdc99e3531f94d8c"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:45.703985 11857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 04:04:45.726014 11873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:45.727774 11857 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260924 04:04:45.727957 11857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "095c9523d1cc48a9baf3fbd40e3a190b"
format_stamp: "Formatted at 2026-09-24 04:04:45 on dist-test-slave-zcsk"
server_key: "33fc825dd2bfd76598f00ae619c5162b"
server_key_iv: "71231bf843efc924bdc99e3531f94d8c"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:45.728121 11857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:04:45.745271 11857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:04:45.749260 11857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:04:45.749468 11857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:04:45.750223 11857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:04:45.751327 11857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:04:45.751398 11857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:45.751452 11857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:04:45.751478 11857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:45.762773 11857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44115
I20260924 04:04:45.764082 11857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:04:45.764035 11986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44115 every 8 connection(s)
I20260924 04:04:45.766440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 11857
I20260924 04:04:45.766599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:04:45.766919 392 external_mini_cluster.cc:1428] Setting key 19d6a877f895fd4fb2da20cc33ef3c01
Sep 24 04:04:45 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:04:45.782918 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:45.766952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51189 (local address 127.0.98.62:35003)
0924 04:04:45.767145 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:45.767150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:45.768409 (+ 1259us) server_negotiation.cc:407] Connection header received
0924 04:04:45.769829 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:45.769832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:45.769886 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:45.769932 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:45.771733 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:45.772591 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:45.774439 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:45.774805 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:45.774899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:45.778531 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:45.778536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:45.778539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:45.778576 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:45.780351 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:45.781055 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:45.781059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:45.781061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:45.781149 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:45.781585 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:45.781588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:45.781590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:45.781820 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:45.781885 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:45.782618 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 04:04:45.782753 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:04:45.784547 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:45.767609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35003 (local address 127.0.98.2:51189)
0924 04:04:45.768244 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:45.768310 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:45.769545 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:45.770363 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:45.770376 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:45.770774 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:04:45.771499 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:45.771521 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:45.772770 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:45.772773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:45.774268 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:45.774281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:45.775158 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:45.775163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:45.775491 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:45.776526 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:04:45.776551 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:04:45.778269 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:04:45.780512 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:45.780545 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:45.780566 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:45.780937 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:45.781299 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:45.781302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:45.781304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:45.781487 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:45.781958 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:04:45.781964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:04:45.782329 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 04:04:45.783087 (+ 758us) client_negotiation.cc:240] Negotiation successful
0924 04:04:45.783448 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":188,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:04:45.789932 11987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35003
I20260924 04:04:45.790246 11987 heartbeater.cc:464] Registering TS with master...
I20260924 04:04:45.790899 11987 heartbeater.cc:511] Master 127.0.98.62:35003 requested a full tablet report, sending...
I20260924 04:04:45.792222 11637 ts_manager.cc:194] Registered new tserver with Master: 095c9523d1cc48a9baf3fbd40e3a190b (127.0.98.2:44115)
I20260924 04:04:45.792982 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:04:45.820568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:04:45 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:04:46.036414 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:04:46.036882 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:04:46.036988 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:04:46.043197 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:04:46.043359 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:04:46.043491 11992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:04:46.052057 11992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:04:46.054072 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:04:46.055598 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:04:46.079282 11998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:46.080098 11997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:04:46.082041 12000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:04:46.104348 11992 server_base.cc:1034] running on GCE node
I20260924 04:04:46.104992 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:04:46.105970 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:04:46.121505 11992 hybrid_clock.cc:648] HybridClock initialized: now 1790222686121384 us; error 128 us; skew 500 ppm
Sep 24 04:04:46 dist-test-slave-zcsk krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:04:46.129449 11992 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:04:46.130937 11992 webserver.cc:460] Webserver started at http://127.0.98.3:43113/ using document root <none> and password file <none>
I20260924 04:04:46.131796 11992 fs_manager.cc:359] Metadata directory not provided
I20260924 04:04:46.131980 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:04:46.132339 11992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:04:46 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:04:46.152559 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "df33146a80504189aed51065c89ded81"
format_stamp: "Formatted at 2026-09-24 04:04:46 on dist-test-slave-zcsk"
server_key: "7481ab72f26d2a6ee13f21c4ad60acc1"
server_key_iv: "2cac63f53def285b99a27848857d4476"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:46.153398 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df33146a80504189aed51065c89ded81"
format_stamp: "Formatted at 2026-09-24 04:04:46 on dist-test-slave-zcsk"
server_key: "7481ab72f26d2a6ee13f21c4ad60acc1"
server_key_iv: "2cac63f53def285b99a27848857d4476"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:46.158723 11992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 04:04:46.175940 12007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:46.177954 11992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:04:46.178200 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "df33146a80504189aed51065c89ded81"
format_stamp: "Formatted at 2026-09-24 04:04:46 on dist-test-slave-zcsk"
server_key: "7481ab72f26d2a6ee13f21c4ad60acc1"
server_key_iv: "2cac63f53def285b99a27848857d4476"
server_key_version: "kuduclusterkey@0"
I20260924 04:04:46.178434 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:04:46.195874 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:04:46.199834 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:04:46.200201 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:04:46.200968 11992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:04:46.202224 11992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:04:46.202339 11992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:46.202436 11992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:04:46.202503 11992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:04:46.214367 11992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38073
I20260924 04:04:46.215648 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:04:46.221606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 11992
I20260924 04:04:46.221725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:04:46.221999 392 external_mini_cluster.cc:1428] Setting key 5eab8158d8470044cb150bee874a86eb
I20260924 04:04:46.217365 12120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38073 every 8 connection(s)
Sep 24 04:04:46 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:04:46.241202 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.220849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60823 (local address 127.0.98.62:35003)
0924 04:04:46.221040 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:46.221046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:46.224830 (+ 3784us) server_negotiation.cc:407] Connection header received
0924 04:04:46.226170 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:46.226173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:46.226225 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:46.226272 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:46.228069 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.228899 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.230719 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.231040 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.231112 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.236502 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:46.236506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:46.236509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:46.236545 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:46.238292 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:46.239137 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:46.239141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:46.239143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:46.239250 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:46.239768 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:46.239772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:46.239773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:46.240044 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:46.240128 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:46.240895 (+ 767us) server_negotiation.cc:299] Negotiation successful
0924 04:04:46.241033 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 04:04:46.243338 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.219587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35003 (local address 127.0.98.3:60823)
0924 04:04:46.224669 (+ 5082us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:46.224736 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:46.225839 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:46.226796 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:46.226806 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:46.227175 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:04:46.227865 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.227885 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.229192 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.229195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.230576 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.230587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.231321 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.231324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.231536 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.232271 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:04:46.232292 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:04:46.234064 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:04:46.238458 (+ 4394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:46.238465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:46.238488 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:46.238968 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:46.239390 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:04:46.239393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:04:46.239395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:04:46.239649 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:04:46.240200 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:04:46.240208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:04:46.240604 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 04:04:46.241805 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0924 04:04:46.242193 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4830,"thread_start_us":152,"threads_started":1}
I20260924 04:04:46.245057 12121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35003
I20260924 04:04:46.245340 12121 heartbeater.cc:464] Registering TS with master...
I20260924 04:04:46.245961 12121 heartbeater.cc:511] Master 127.0.98.62:35003 requested a full tablet report, sending...
I20260924 04:04:46.248916 11637 ts_manager.cc:194] Registered new tserver with Master: df33146a80504189aed51065c89ded81 (127.0.98.3:38073)
I20260924 04:04:46.249742 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60823
I20260924 04:04:46.260226 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:04:46.273612 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.262644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.0.98.62:35003)
0924 04:04:46.262760 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:46.262765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:46.263347 (+ 582us) server_negotiation.cc:407] Connection header received
0924 04:04:46.263539 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:46.263542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:46.263590 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:46.263637 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:04:46.264645 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.265407 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.267218 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.267571 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.267668 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.269901 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:04:46.269905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:04:46.269908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:04:46.269940 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:04:46.271721 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:46.272300 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:46.272304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:46.272306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:46.272396 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:04:46.272676 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:04:46.272679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:04:46.272681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:04:46.272889 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:04:46.272994 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:46.273312 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 04:04:46.273433 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:04:46.277817 11637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:04:46.282333 11637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:04:46.307016 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.296415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38073 (local address 127.0.0.1:38570)
0924 04:04:46.299377 (+ 2962us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:46.299388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:46.299447 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:46.300370 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:46.300376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:46.300397 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:46.300533 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.300538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.302886 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.302889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.305045 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.305056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.306385 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.306388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.306525 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.306582 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 04:04:46.306697 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:04:46.306839 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2906,"thread_start_us":116,"threads_started":1}
I20260924 04:04:46.308974 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.296843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.0.98.3:38073)
0924 04:04:46.299701 (+ 2858us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:46.299712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:46.299734 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:04:46.299807 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:46.299812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:46.300060 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:46.300203 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:46.301647 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.302761 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.305273 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.306179 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.306281 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.308650 (+ 2369us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:46.308708 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 04:04:46.308810 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2750,"thread_start_us":116,"threads_started":1}
I20260924 04:04:46.313099 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.297611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44115 (local address 127.0.0.1:49242)
0924 04:04:46.301443 (+ 3832us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:46.301455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:46.301535 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:46.308814 (+ 7279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:46.308817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:46.308835 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:46.308966 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.308973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.310325 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.310328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.311592 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.311604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.312668 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.312672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.312758 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.312797 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:04:46.312850 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:04:46.312927 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3776,"mutex_wait_us":48,"thread_start_us":81,"threads_started":1}
I20260924 04:04:46.312502 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 96ad4d67e940471a89c2d13e5894944d (DEFAULT_TABLE table=test-table [id=3012f9485e224f12b7ea3eb433f8cdac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:04:46.313165 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.294469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36725 (local address 127.0.0.1:49280)
0924 04:04:46.298975 (+ 4506us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:46.299038 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:46.299175 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:46.308826 (+ 9651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:46.308840 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:46.308873 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:46.309017 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.309023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.310434 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.310437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.311689 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.311700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.312766 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.312770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.312870 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.312899 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:04:46.312946 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:04:46.313014 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4320,"mutex_wait_us":42,"spinlock_wait_cycles":2176,"thread_start_us":107,"threads_started":1}
I20260924 04:04:46.313755 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.294618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.0.98.1:36725)
0924 04:04:46.308142 (+ 13524us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:46.308150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:46.308179 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:04:46.308265 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:46.308271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:46.308496 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:46.308665 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:46.309152 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.310303 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.311854 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.312634 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.313337 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.313487 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:46.313537 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:04:46.313626 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13398,"thread_start_us":116,"threads_started":1}
I20260924 04:04:46.313863 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.297816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.0.98.2:44115)
0924 04:04:46.308133 (+ 10317us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:46.308141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:46.308168 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:04:46.308250 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:46.308256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:46.308511 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:46.308667 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:46.309111 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.310189 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.311733 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.312533 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.313448 (+ 915us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.313595 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:46.313642 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 04:04:46.313734 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10195,"thread_start_us":108,"threads_started":1}
I20260924 04:04:46.315225 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ad4d67e940471a89c2d13e5894944d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:04:46.316690 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 96ad4d67e940471a89c2d13e5894944d (DEFAULT_TABLE table=test-table [id=3012f9485e224f12b7ea3eb433f8cdac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:04:46.316690 11785 tablet_service.cc:1467] Processing CreateTablet for tablet 96ad4d67e940471a89c2d13e5894944d (DEFAULT_TABLE table=test-table [id=3012f9485e224f12b7ea3eb433f8cdac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:04:46.317773 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ad4d67e940471a89c2d13e5894944d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:04:46.317773 11785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ad4d67e940471a89c2d13e5894944d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:04:46.325989 12140 tablet_bootstrap.cc:492] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Bootstrap starting.
I20260924 04:04:46.330994 12140 tablet_bootstrap.cc:654] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Neither blocks nor log segments found. Creating new log.
I20260924 04:04:46.332146 12140 log.cc:824] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Log is configured to *not* fsync() on all Append() calls
I20260924 04:04:46.334673 12140 tablet_bootstrap.cc:492] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: No bootstrap required, opened a new log
I20260924 04:04:46.334882 12140 ts_tablet_manager.cc:1397] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 04:04:46.336143 12142 tablet_bootstrap.cc:492] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Bootstrap starting.
I20260924 04:04:46.337126 12143 tablet_bootstrap.cc:492] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74: Bootstrap starting.
I20260924 04:04:46.338603 12142 tablet_bootstrap.cc:654] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Neither blocks nor log segments found. Creating new log.
I20260924 04:04:46.339036 12140 raft_consensus.cc:348] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.339286 12143 tablet_bootstrap.cc:654] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74: Neither blocks nor log segments found. Creating new log.
I20260924 04:04:46.339321 12140 raft_consensus.cc:374] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:04:46.339371 12140 raft_consensus.cc:729] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df33146a80504189aed51065c89ded81, State: Initialized, Role: FOLLOWER
I20260924 04:04:46.339464 12142 log.cc:824] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:04:46.340103 12143 log.cc:824] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74: Log is configured to *not* fsync() on all Append() calls
I20260924 04:04:46.341369 12140 consensus_queue.cc:260] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.342648 12140 ts_tablet_manager.cc:1428] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260924 04:04:46.343076 12121 heartbeater.cc:503] Master 127.0.98.62:35003 was elected leader, sending a full tablet report...
I20260924 04:04:46.351593 12142 tablet_bootstrap.cc:492] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: No bootstrap required, opened a new log
I20260924 04:04:46.351845 12142 ts_tablet_manager.cc:1397] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260924 04:04:46.352952 12143 tablet_bootstrap.cc:492] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74: No bootstrap required, opened a new log
I20260924 04:04:46.353181 12143 ts_tablet_manager.cc:1397] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260924 04:04:46.356681 12142 raft_consensus.cc:348] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.356999 12142 raft_consensus.cc:374] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:04:46.357059 12142 raft_consensus.cc:729] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095c9523d1cc48a9baf3fbd40e3a190b, State: Initialized, Role: FOLLOWER
I20260924 04:04:46.357614 12142 consensus_queue.cc:260] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.358093 12143 raft_consensus.cc:348] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.358353 12143 raft_consensus.cc:374] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:04:46.358418 12143 raft_consensus.cc:729] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b33d286b2644b7bec41d30f3322c74, State: Initialized, Role: FOLLOWER
I20260924 04:04:46.358745 12142 ts_tablet_manager.cc:1428] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260924 04:04:46.359109 11987 heartbeater.cc:503] Master 127.0.98.62:35003 was elected leader, sending a full tablet report...
I20260924 04:04:46.359328 12143 consensus_queue.cc:260] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.364656 12143 ts_tablet_manager.cc:1428] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260924 04:04:46.365083 11851 heartbeater.cc:503] Master 127.0.98.62:35003 was elected leader, sending a full tablet report...
Sep 24 04:04:46 dist-test-slave-zcsk krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:04:46.429512 11852 tablet.cc:2369] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:04:46.474620 12122 tablet.cc:2369] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:04:46.527091 11988 tablet.cc:2369] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:04:46.618312 12146 raft_consensus.cc:482] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:04:46.618571 12146 raft_consensus.cc:504] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.620026 12146 leader_election.cc:288] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b33d286b2644b7bec41d30f3322c74 (127.0.98.1:36725), 095c9523d1cc48a9baf3fbd40e3a190b (127.0.98.2:44115)
I20260924 04:04:46.625345 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.620271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36725 (local address 127.0.98.3:35087)
0924 04:04:46.620395 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:46.620412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:46.620523 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:46.621396 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:46.621399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:46.621418 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:46.621541 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.621548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.622571 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.622574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.623938 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.623949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.624937 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.624940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.625043 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.625064 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:04:46.625129 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:04:46.625199 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":12}
I20260924 04:04:46.625814 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.621013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35087 (local address 127.0.98.1:36725)
0924 04:04:46.621144 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:46.621149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:46.621166 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:04:46.621222 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:46.621225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:46.621276 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:46.621358 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:46.621679 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.622442 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.624092 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.624688 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.624792 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.624877 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:46.625616 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 04:04:46.625697 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:04:46.626473 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ad4d67e940471a89c2d13e5894944d" candidate_uuid: "df33146a80504189aed51065c89ded81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b33d286b2644b7bec41d30f3322c74" is_pre_election: true
I20260924 04:04:46.626714 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.621550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46757 (local address 127.0.98.2:44115)
0924 04:04:46.621700 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:46.621705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:46.621721 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:04:46.621772 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:46.621775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:46.621822 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:46.621893 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:04:46.622374 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.623097 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.624669 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.625297 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.625376 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.625436 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:46.626480 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0924 04:04:46.626586 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:04:46.626883 11805 raft_consensus.cc:2454] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df33146a80504189aed51065c89ded81 in term 0.
I20260924 04:04:46.627404 12008 leader_election.cc:302] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b33d286b2644b7bec41d30f3322c74, df33146a80504189aed51065c89ded81; no voters:
I20260924 04:04:46.627673 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.620581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44115 (local address 127.0.98.3:46757)
0924 04:04:46.620897 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:46.620911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:04:46.620996 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:04:46.622104 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:04:46.622107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:04:46.622125 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:04:46.622246 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.622252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.623238 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.623241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.624513 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:04:46.624535 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.625804 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.625808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:04:46.626333 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.626359 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:04:46.627116 (+ 757us) client_negotiation.cc:240] Negotiation successful
0924 04:04:46.627325 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260924 04:04:46.627768 12146 raft_consensus.cc:2781] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:04:46.627923 12146 raft_consensus.cc:482] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:04:46.627982 12146 raft_consensus.cc:3037] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:04:46.628618 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ad4d67e940471a89c2d13e5894944d" candidate_uuid: "df33146a80504189aed51065c89ded81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" is_pre_election: true
I20260924 04:04:46.628928 11930 raft_consensus.cc:2454] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df33146a80504189aed51065c89ded81 in term 0.
I20260924 04:04:46.629335 12146 raft_consensus.cc:504] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.629694 12146 leader_election.cc:288] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [CANDIDATE]: Term 1 election: Requested vote from peers 10b33d286b2644b7bec41d30f3322c74 (127.0.98.1:36725), 095c9523d1cc48a9baf3fbd40e3a190b (127.0.98.2:44115)
I20260924 04:04:46.630049 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ad4d67e940471a89c2d13e5894944d" candidate_uuid: "df33146a80504189aed51065c89ded81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b33d286b2644b7bec41d30f3322c74"
I20260924 04:04:46.630157 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ad4d67e940471a89c2d13e5894944d" candidate_uuid: "df33146a80504189aed51065c89ded81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095c9523d1cc48a9baf3fbd40e3a190b"
I20260924 04:04:46.630182 11805 raft_consensus.cc:3037] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:04:46.630276 11930 raft_consensus.cc:3037] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:04:46.631397 11805 raft_consensus.cc:2454] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df33146a80504189aed51065c89ded81 in term 1.
I20260924 04:04:46.631593 11930 raft_consensus.cc:2454] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df33146a80504189aed51065c89ded81 in term 1.
I20260924 04:04:46.631750 12008 leader_election.cc:302] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b33d286b2644b7bec41d30f3322c74, df33146a80504189aed51065c89ded81; no voters:
I20260924 04:04:46.632102 12146 raft_consensus.cc:2781] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:04:46.632341 12146 raft_consensus.cc:686] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 1 LEADER]: Becoming Leader. State: Replica: df33146a80504189aed51065c89ded81, State: Running, Role: LEADER
I20260924 04:04:46.632872 12146 consensus_queue.cc:237] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } }
I20260924 04:04:46.638255 11637 catalog_manager.cc:5495] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 reported cstate change: term changed from 0 to 1, leader changed from <none> to df33146a80504189aed51065c89ded81 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "df33146a80504189aed51065c89ded81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df33146a80504189aed51065c89ded81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 } health_report { overall_health: UNKNOWN } } }
I20260924 04:04:46.677806 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.674090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.0.98.3:38073)
0924 04:04:46.674218 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:04:46.674222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:04:46.674319 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:04:46.674510 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:04:46.674512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:04:46.674573 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:04:46.674663 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:04:46.674899 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.675412 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.676337 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:04:46.676558 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:04:46.676645 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:04:46.676940 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:04:46.677403 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:04:46.677521 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:04:46.677594 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 04:04:46.677652 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:04:46.684834 11805 raft_consensus.cc:1261] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 1 FOLLOWER]: Refusing update from remote peer df33146a80504189aed51065c89ded81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:04:46.684857 11930 raft_consensus.cc:1261] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 1 FOLLOWER]: Refusing update from remote peer df33146a80504189aed51065c89ded81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:04:46.685695 12152 consensus_queue.cc:1035] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b33d286b2644b7bec41d30f3322c74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:04:46.685986 12146 consensus_queue.cc:1035] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "095c9523d1cc48a9baf3fbd40e3a190b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:04:46.698563 12164 mvcc.cc:205] Tried to move back new op lower bound from 7332752124653101056 to 7332752124452089856. Current Snapshot: MvccSnapshot[applied={T|T < 7332752124653101056}]
I20260924 04:04:46.701223 12162 mvcc.cc:205] Tried to move back new op lower bound from 7332752124653101056 to 7332752124452089856. Current Snapshot: MvccSnapshot[applied={T|T < 7332752124653101056 or (T in {7332752124653101056})}]
I20260924 04:04:46.707680 11637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:04:46.707932 11637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:04:46.710784 11637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb356f5659b48e09cf8c4f2ad54b847: Sending DeleteTablet for 3 replicas of tablet 96ad4d67e940471a89c2d13e5894944d
I20260924 04:04:46.711757 11785 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ad4d67e940471a89c2d13e5894944d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:04:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49280
I20260924 04:04:46.711750 11921 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ad4d67e940471a89c2d13e5894944d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:04:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49242
I20260924 04:04:46.712607 12172 tablet_replica.cc:330] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74: stopping tablet replica
I20260924 04:04:46.712639 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ad4d67e940471a89c2d13e5894944d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:04:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38570
I20260924 04:04:46.712978 12172 raft_consensus.cc:2229] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:04:46.713348 12172 raft_consensus.cc:2258] T 96ad4d67e940471a89c2d13e5894944d P 10b33d286b2644b7bec41d30f3322c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:04:46.713739 12173 tablet_replica.cc:330] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: stopping tablet replica
I20260924 04:04:46.714072 12173 raft_consensus.cc:2229] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:04:46.714284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 11718
I20260924 04:04:46.714380 12173 raft_consensus.cc:2258] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:04:46.716329 12173 ts_tablet_manager.cc:1905] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:04:46.719108 12173 ts_tablet_manager.cc:1918] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:04:46.719203 12173 log.cc:1196] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/96ad4d67e940471a89c2d13e5894944d
I20260924 04:04:46.719501 12173 ts_tablet_manager.cc:1939] T 96ad4d67e940471a89c2d13e5894944d P 095c9523d1cc48a9baf3fbd40e3a190b: Deleting consensus metadata
I20260924 04:04:46.720928 12174 tablet_replica.cc:330] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: stopping tablet replica
I20260924 04:04:46.721246 12174 raft_consensus.cc:2229] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:04:46.721644 12174 raft_consensus.cc:2258] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:04:46.722431 11621 catalog_manager.cc:4841] TS 095c9523d1cc48a9baf3fbd40e3a190b (127.0.98.2:44115): tablet 96ad4d67e940471a89c2d13e5894944d (table test-table [id=3012f9485e224f12b7ea3eb433f8cdac]) successfully deleted
I20260924 04:04:46.723184 12174 ts_tablet_manager.cc:1905] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:04:46.724417 11621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36725 (error 108)
I20260924 04:04:46.725283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 11857
I20260924 04:04:46.726619 12174 ts_tablet_manager.cc:1918] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:04:46.726706 12174 log.cc:1196] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/96ad4d67e940471a89c2d13e5894944d
I20260924 04:04:46.726996 12174 ts_tablet_manager.cc:1939] T 96ad4d67e940471a89c2d13e5894944d P df33146a80504189aed51065c89ded81: Deleting consensus metadata
I20260924 04:04:46.732190 11622 catalog_manager.cc:4841] TS df33146a80504189aed51065c89ded81 (127.0.98.3:38073): tablet 96ad4d67e940471a89c2d13e5894944d (table test-table [id=3012f9485e224f12b7ea3eb433f8cdac]) successfully deleted
I20260924 04:04:46.733072 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:04:46.732471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36725 (local address 127.0.0.1:49284)
0924 04:04:46.732681 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 04:04:46.732952 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260924 04:04:46.733330 11621 catalog_manager.cc:4584] TS 10b33d286b2644b7bec41d30f3322c74 (127.0.98.1:36725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96ad4d67e940471a89c2d13e5894944d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36725: connect: Connection refused (error 111)
I20260924 04:04:46.733610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 11992
I20260924 04:04:46.738983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 11605
I20260924 04:04:46.746469 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:04:47.081578 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:04:47.081712 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:04:47.427095 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:04:47.427 UTC [11191] LOG: received smart shutdown request
2026-09-24 04:04:47.430 UTC [11191] LOG: background worker "logical replication launcher" (PID 11200) exited with exit code 1
2026-09-24 04:04:47.430 UTC [11195] LOG: shutting down
2026-09-24 04:04:47.597 UTC [11191] LOG: database system is shut down
2026-09-24T04:04:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45956 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:04:47 dist-test-slave-zcsk krb5kdc[12184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:04:47 dist-test-slave-zcsk krb5kdc[12184](info): set up 2 sockets
Sep 24 04:04:47 dist-test-slave-zcsk krb5kdc[12184](info): commencing operation
krb5kdc: starting...
W20260924 04:04:48.663760 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:04:48 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:04:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:04:48.834391 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskeFjGAy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:04:51.364 UTC [12214] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:04:51.364 UTC [12214] LOG: listening on IPv4 address "127.0.98.20", port 58593
2026-09-24 04:04:51.367 UTC [12214] LOG: listening on Unix socket "/tmp/.s.PGSQL.58593"
2026-09-24 04:04:51.387 UTC [12217] LOG: database system was shut down at 2026-09-24 04:04:49 UTC
2026-09-24 04:04:51.390 UTC [12214] LOG: database system is ready to accept connections
I20260924 04:04:52.267729 392 mini_postgres.cc:96] Postgres bound to 58593
2026-09-24 04:04:52.273 UTC [12225] FATAL: database "slave" does not exist
127.0.98.20:58593 - accepting connections
I20260924 04:04:52.274482 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:04:52.289492 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:04:52.792716 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:04:52.792802 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-admin
2026-09-24 04:04:53,085 [I] DB FLAVOR :POSTGRES
2026-09-24 04:04:53,085 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:04:53,086 [I] Checking connection..
2026-09-24 04:04:53,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:04:53,316 [I] Checking connection passed.
2026-09-24 04:04:53,316 [I] --------- Verifying version history table ---------
2026-09-24 04:04:53,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:04:53,574 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:04:53,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:04:53,838 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:04:53,838 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:04:53,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:04:54.095 UTC [12289] WARNING: there is no transaction in progress
2026-09-24 04:04:54,101 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:04:54,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:04:54,362 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:04:54,362 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:04:54,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:04:54,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:04:54,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zcsk','N') ;"
2026-09-24 04:04:55,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:04:55,366 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:04:55,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:04:55,603 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:04:55,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:04:55,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:04:56,099 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:04:56,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:04:57.140 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.158 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.176 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.192 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.211 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.229 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.290 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.297 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.313 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.324 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.335 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.350 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.360 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.367 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.375 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:57.383 UTC [12397] WARNING: there is no transaction in progress
2026-09-24 04:04:58,105 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:04:58,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:04:58,333 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:04:58,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:04:58,598 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:04:58,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:04:58,847 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:04:58,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:04:59,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:04:59,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zcsk';"
2026-09-24 04:04:59,576 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:04:59,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:04:59,832 [I] DB_PATCHES have already been applied
I20260924 04:04:59.837664 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskeFjGAy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:04:59.837729 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:04:59.841023 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52975
Sep 24, 2026 4:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:05:00.857327 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.583s user 0.002s sys 0.007s
Sep 24, 2026 4:05:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:05:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 4:05:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:05:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:05:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:05:23.345664 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:05:23 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222723, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:05:23.442826 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:05:23.452260 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:05:24.308365 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:05:24.308435 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms
I20260924 04:05:24.308449 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58593
2026-09-24 04:05:24,462 [I] DB FLAVOR :POSTGRES
2026-09-24 04:05:24,463 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:05:24,463 [I] Checking connection
2026-09-24 04:05:24,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:05:24,702 [I] connection success
2026-09-24 04:05:24,702 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:05:24,702 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:05:24,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:05:24,941 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:05:24,941 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:05:24,941 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:05:24,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:05:25,187 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:05:25.646941 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:05:25.853879 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:05:25.897612 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:05:25 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222723, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:26.300490 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:05:26.300657 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskeFjGAy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskeFjGAy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:05:26.300683 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:05:26.304841 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12577
I20260924 04:05:26.304919 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33167
Sep 24, 2026 4:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:05:28.315208 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.872s user 0.012s sys 0.126s
I20260924 04:05:28.315328 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:05:28.315403 392 mini_ranger_kms.cc:401] 127.0.98.20:33167/kms/v1/keys
Sep 24 04:05:28 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790222728, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:05:28 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222728, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:05:29 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222728, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:05:31 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:05:31.566702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36933
--ranger_config_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33167/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:31.708072 12628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:31.708371 12628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:31.708428 12628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:31.712553 12628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:31.712628 12628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:31.712656 12628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:31.712677 12628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:31.712707 12628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:31.718526 12628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33167/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36933
--ranger_config_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:31.719676 12628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:31.720769 12628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:31.730657 12634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:31.730657 12636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:31.730657 12633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:31.731055 12628 server_base.cc:1034] running on GCE node
I20260924 04:05:31.731655 12628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:31.732900 12628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:31.734073 12628 hybrid_clock.cc:648] HybridClock initialized: now 1790222731734044 us; error 51 us; skew 500 ppm
Sep 24 04:05:31 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:31.737246 12628 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:05:31.738405 12628 webserver.cc:460] Webserver started at http://127.0.98.62:40991/ using document root <none> and password file <none>
I20260924 04:05:31.739126 12628 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:31.739202 12628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:31.739430 12628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:05:31 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:31.794839 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "6306ce75e76c4e0fb97dfba94671ecf8"
format_stamp: "Formatted at 2026-09-24 04:05:31 on dist-test-slave-zcsk"
server_key: "f928b00afc349f8dccd15dfe5b889589"
server_key_iv: "27b11610a7f14fb266f531ed79db9ab1"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:31.795578 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "6306ce75e76c4e0fb97dfba94671ecf8"
format_stamp: "Formatted at 2026-09-24 04:05:31 on dist-test-slave-zcsk"
server_key: "f928b00afc349f8dccd15dfe5b889589"
server_key_iv: "27b11610a7f14fb266f531ed79db9ab1"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:31.800631 12628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 04:05:31.818766 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:31.820492 12628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:05:31.820647 12628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "6306ce75e76c4e0fb97dfba94671ecf8"
format_stamp: "Formatted at 2026-09-24 04:05:31 on dist-test-slave-zcsk"
server_key: "f928b00afc349f8dccd15dfe5b889589"
server_key_iv: "27b11610a7f14fb266f531ed79db9ab1"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:31.820798 12628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:31.841790 12628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:31.845232 12628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:31.845422 12628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:31.853775 12628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36933
I20260924 04:05:31.853775 12695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36933 every 8 connection(s)
I20260924 04:05:31.855010 12628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:05:31.855433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 12628
I20260924 04:05:31.855568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:05:31.855870 392 external_mini_cluster.cc:1428] Setting key d3029a20d61eb5a7e6fb77d471a2bfa3
I20260924 04:05:31.858896 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:31.865422 12696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Bootstrap starting.
Sep 24 04:05:31 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:31.868189 12696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:31.869355 12696 log.cc:824] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:31.871769 12696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: No bootstrap required, opened a new log
I20260924 04:05:31.872341 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:31.857359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.0.98.62:36933)
0924 04:05:31.857830 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:31.857849 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:31.857894 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:05:31.858776 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:31.858800 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:31.859254 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:31.859638 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:31.860525 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:31.861969 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:31.863110 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:31.863769 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:31.863959 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:31.866949 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:31.866955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:31.866974 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:31.867015 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:31.869020 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:31.869575 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:31.869581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:31.869587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:31.869722 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:31.870009 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:31.870018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:31.870019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:31.870517 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:31.870709 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:31.871041 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 04:05:31.871327 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":148,"threads_started":1}
I20260924 04:05:31.875313 12696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } }
I20260924 04:05:31.875556 12696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:31.875593 12696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6306ce75e76c4e0fb97dfba94671ecf8, State: Initialized, Role: FOLLOWER
I20260924 04:05:31.876081 12696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } }
I20260924 04:05:31.876219 12696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:05:31.876299 12696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:05:31.876401 12696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:31.877347 12696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } }
I20260924 04:05:31.877686 12696 leader_election.cc:302] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6306ce75e76c4e0fb97dfba94671ecf8; no voters:
I20260924 04:05:31.877979 12696 leader_election.cc:288] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:05:31.878087 12701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:05:31.878271 12701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 1 LEADER]: Becoming Leader. State: Replica: 6306ce75e76c4e0fb97dfba94671ecf8, State: Running, Role: LEADER
I20260924 04:05:31.878636 12701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } }
I20260924 04:05:31.879295 12696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:05:31.879819 12701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6306ce75e76c4e0fb97dfba94671ecf8. Latest consensus state: current_term: 1 leader_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } } }
I20260924 04:05:31.880054 12701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:31.880234 12702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } } }
I20260924 04:05:31.880398 12702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:31.880470 12709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:05:31.883788 12709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:05:31.884277 12696 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:05:31.898360 12709 catalog_manager.cc:1347] Generated new cluster ID: 82971a98b03d4f16977eb12b663fc706
I20260924 04:05:31.898442 12709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:05:31.917948 12709 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:05:31.919423 12709 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:05:31.928414 12709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Generated new TSK 0
I20260924 04:05:31.929366 12709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:05:32.613322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36933
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:32.734221 12741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:32.734542 12741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:32.734659 12741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:32.739305 12741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:32.739380 12741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:32.739463 12741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:05:32.745218 12741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:32.746476 12741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:32.747563 12741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:32.757434 12747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:32.757387 12746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:32.757387 12749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:32.758313 12741 server_base.cc:1034] running on GCE node
I20260924 04:05:32.758759 12741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:32.759449 12741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:32.760636 12741 hybrid_clock.cc:648] HybridClock initialized: now 1790222732760592 us; error 48 us; skew 500 ppm
Sep 24 04:05:32 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:32.764874 12741 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:05:32.766194 12741 webserver.cc:460] Webserver started at http://127.0.98.1:42397/ using document root <none> and password file <none>
I20260924 04:05:32.767009 12741 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:32.767108 12741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:32.767359 12741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:05:32 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:32.786063 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd0a9b3bb416458189c3668470f469c4"
format_stamp: "Formatted at 2026-09-24 04:05:32 on dist-test-slave-zcsk"
server_key: "0a6fd10db569dde34fd799b44dc0e956"
server_key_iv: "1a546302124ac8bec0ddf2f0437c063f"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:32.786859 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd0a9b3bb416458189c3668470f469c4"
format_stamp: "Formatted at 2026-09-24 04:05:32 on dist-test-slave-zcsk"
server_key: "0a6fd10db569dde34fd799b44dc0e956"
server_key_iv: "1a546302124ac8bec0ddf2f0437c063f"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:32.792013 12741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 04:05:32.816342 12758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:32.817845 12741 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260924 04:05:32.817991 12741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "dd0a9b3bb416458189c3668470f469c4"
format_stamp: "Formatted at 2026-09-24 04:05:32 on dist-test-slave-zcsk"
server_key: "0a6fd10db569dde34fd799b44dc0e956"
server_key_iv: "1a546302124ac8bec0ddf2f0437c063f"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:32.818145 12741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:32.846328 12741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:32.849400 12741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:32.849576 12741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:32.850194 12741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:32.851033 12741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:32.851089 12741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:32.851145 12741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:32.851173 12741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:32.861711 12741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44669
I20260924 04:05:32.862897 12741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:05:32.863257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 12741
I20260924 04:05:32.863380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:05:32.863670 392 external_mini_cluster.cc:1428] Setting key 2045fb279f43f7c965fdb39e67eac37c
I20260924 04:05:32.867425 12871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44669 every 8 connection(s)
Sep 24 04:05:32 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:32.886371 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:32.870973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36933 (local address 127.0.98.1:44775)
0924 04:05:32.871617 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:32.871680 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:32.872758 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:32.873336 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:32.873347 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:32.873714 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:32.874378 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:32.874397 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:32.875548 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:32.875552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:32.876987 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:32.876998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:32.877782 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:32.877786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:32.877987 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:32.878812 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:32.878833 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:32.880645 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:32.882865 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:32.882872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:32.882891 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:32.883252 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:32.883575 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:32.883578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:32.883580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:32.883786 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:32.884308 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:32.884315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:32.884689 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 04:05:32.884891 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 04:05:32.885245 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":162,"threads_started":1}
I20260924 04:05:32.887187 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:32.872203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44775 (local address 127.0.98.62:36933)
0924 04:05:32.872494 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:32.872502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:32.872585 (+ 83us) server_negotiation.cc:407] Connection header received
0924 04:05:32.872992 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:32.872996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:32.873051 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:32.873156 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:32.874541 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:32.875408 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:32.877136 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:32.877497 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:32.877605 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:32.880863 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:32.880867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:32.880870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:32.880898 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:32.882721 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:32.883371 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:32.883374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:32.883376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:32.883467 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:32.883895 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:32.883898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:32.883927 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:32.884171 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:32.884256 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:32.886883 (+ 2627us) server_negotiation.cc:299] Negotiation successful
0924 04:05:32.887019 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260924 04:05:32.888355 12872 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36933
I20260924 04:05:32.888643 12872 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:32.889292 12872 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
I20260924 04:05:32.891105 12660 ts_manager.cc:194] Registered new tserver with Master: dd0a9b3bb416458189c3668470f469c4 (127.0.98.1:44669)
I20260924 04:05:32.892705 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44775
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:05:32.919569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36933
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:05:32 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790222732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:05:33.080457 12880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:33.080871 12880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:33.080952 12880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:33.085268 12880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:33.085347 12880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:33.085435 12880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:05:33.093657 12880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:33.095111 12880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:33.096541 12880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:33.109071 12886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:33.109551 12880 server_base.cc:1034] running on GCE node
W20260924 04:05:33.109097 12889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:33.109274 12887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:33.110407 12880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:33.111281 12880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:33.112488 12880 hybrid_clock.cc:648] HybridClock initialized: now 1790222733112462 us; error 40 us; skew 500 ppm
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:33.116147 12880 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:05:33.117291 12880 webserver.cc:460] Webserver started at http://127.0.98.2:35645/ using document root <none> and password file <none>
I20260924 04:05:33.118001 12880 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:33.118101 12880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:33.118351 12880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:33.138917 12880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "03cb21bee90c42c78f51dda72f8f856c"
format_stamp: "Formatted at 2026-09-24 04:05:33 on dist-test-slave-zcsk"
server_key: "7e2192cc54d471e1f74cfabafd29a42a"
server_key_iv: "aa9872ef76bbe0bc29e381edb31c9749"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:33.139696 12880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "03cb21bee90c42c78f51dda72f8f856c"
format_stamp: "Formatted at 2026-09-24 04:05:33 on dist-test-slave-zcsk"
server_key: "7e2192cc54d471e1f74cfabafd29a42a"
server_key_iv: "aa9872ef76bbe0bc29e381edb31c9749"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:33.145748 12880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:05:33.167861 12896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:33.169277 12880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 04:05:33.169390 12880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "03cb21bee90c42c78f51dda72f8f856c"
format_stamp: "Formatted at 2026-09-24 04:05:33 on dist-test-slave-zcsk"
server_key: "7e2192cc54d471e1f74cfabafd29a42a"
server_key_iv: "aa9872ef76bbe0bc29e381edb31c9749"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:33.169523 12880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:33.183454 12880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:33.187004 12880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:33.187220 12880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:33.187985 12880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:33.188908 12880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:33.188957 12880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:33.188997 12880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:33.189034 12880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:33.198055 12880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42333
I20260924 04:05:33.198206 13009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42333 every 8 connection(s)
I20260924 04:05:33.199007 12880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:05:33.208532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 12880
I20260924 04:05:33.208670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:05:33.208990 392 external_mini_cluster.cc:1428] Setting key 540bb8e67efe5bcbdd66d090d7038e00
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:33.218040 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:33.201293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58425 (local address 127.0.98.62:36933)
0924 04:05:33.201433 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:33.201438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:33.202550 (+ 1112us) server_negotiation.cc:407] Connection header received
0924 04:05:33.203809 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:33.203813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:33.203873 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:33.204011 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:33.205556 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:33.206400 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:33.208193 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:33.208515 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:33.208597 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:33.213676 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:33.213680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:33.213683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:33.213715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:33.215471 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:33.216138 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:33.216142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:33.216144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:33.216223 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:33.216590 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:33.216594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:33.216596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:33.216834 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:33.216923 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:33.217686 (+ 763us) server_negotiation.cc:299] Negotiation successful
0924 04:05:33.217830 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:05:33.219985 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:33.201786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36933 (local address 127.0.98.2:58425)
0924 04:05:33.202373 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:33.202437 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:33.203555 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:33.204331 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:33.204341 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:33.204701 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:33.205378 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:33.205399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:33.206559 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:33.206562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:33.208053 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:33.208065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:33.208815 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:33.208820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:33.209038 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:33.210185 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:33.210208 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:33.213454 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:33.215621 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:33.215628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:33.215650 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:33.216041 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:33.216307 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:33.216310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:33.216313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:33.216479 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:33.216995 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:33.217003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:33.217392 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 04:05:33.218496 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0924 04:05:33.218879 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":170,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:05:33.221535 13010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36933
I20260924 04:05:33.221772 13010 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:33.222386 13010 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
I20260924 04:05:33.223485 12660 ts_manager.cc:194] Registered new tserver with Master: 03cb21bee90c42c78f51dda72f8f856c (127.0.98.2:42333)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:05:33.224354 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:05:33.243412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36933
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:33.403688 13015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:33.404069 13015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:33.404143 13015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:33.410246 13015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:33.410337 13015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:33.410439 13015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:05:33.417822 13015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:33.419291 13015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:33.420714 13015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:33.433147 13021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:33.433470 13015 server_base.cc:1034] running on GCE node
W20260924 04:05:33.433173 13023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:33.433799 13020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:33.434331 13015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:33.435176 13015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:33.436362 13015 hybrid_clock.cc:648] HybridClock initialized: now 1790222733436330 us; error 49 us; skew 500 ppm
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:33.439893 13015 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:05:33.441022 13015 webserver.cc:460] Webserver started at http://127.0.98.3:33639/ using document root <none> and password file <none>
I20260924 04:05:33.441706 13015 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:33.441781 13015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:33.442008 13015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:33.457216 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "ea9791a31a734daaa435b46117af4c88"
format_stamp: "Formatted at 2026-09-24 04:05:33 on dist-test-slave-zcsk"
server_key: "a8d47dd1ba4ac6b696601d6ed23911e6"
server_key_iv: "a3ec836426b960670a27b14ae2f80356"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:33.458091 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ea9791a31a734daaa435b46117af4c88"
format_stamp: "Formatted at 2026-09-24 04:05:33 on dist-test-slave-zcsk"
server_key: "a8d47dd1ba4ac6b696601d6ed23911e6"
server_key_iv: "a3ec836426b960670a27b14ae2f80356"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:33.481876 13015 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.002s sys 0.004s
I20260924 04:05:33.504299 13030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:33.509449 13015 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260924 04:05:33.509620 13015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "ea9791a31a734daaa435b46117af4c88"
format_stamp: "Formatted at 2026-09-24 04:05:33 on dist-test-slave-zcsk"
server_key: "a8d47dd1ba4ac6b696601d6ed23911e6"
server_key_iv: "a3ec836426b960670a27b14ae2f80356"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:33.509773 13015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:33.543242 13015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:33.546975 13015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:33.547165 13015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:33.547851 13015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:33.548928 13015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:33.548990 13015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:33.549044 13015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:33.549073 13015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:33.560683 13015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40981
I20260924 04:05:33.560716 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40981 every 8 connection(s)
I20260924 04:05:33.561847 13015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:05:33.563560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 13015
I20260924 04:05:33.563676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:05:33.564004 392 external_mini_cluster.cc:1428] Setting key 82fe57fb9060ec9cbc4a3744f8133bcc
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:33.580411 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:33.566859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51737 (local address 127.0.98.62:36933)
0924 04:05:33.566985 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:33.566990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:33.567005 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:05:33.567722 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:33.567726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:33.567779 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:33.567828 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:33.569477 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:33.570334 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:33.572047 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:33.572357 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:33.572463 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:33.576125 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:33.576130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:33.576133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:33.576165 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:33.577762 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:33.578411 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:33.578415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:33.578417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:33.578529 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:33.578974 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:33.578978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:33.578980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:33.579206 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:33.579280 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:33.580118 (+ 838us) server_negotiation.cc:299] Negotiation successful
0924 04:05:33.580246 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:05:33.582115 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:33.565695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36933 (local address 127.0.98.3:51737)
0924 04:05:33.566318 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:33.566387 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:33.567475 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:33.568270 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:33.568280 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:33.568649 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:33.569312 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:33.569331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:33.570469 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:33.570473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:33.571856 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:33.571866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:33.572522 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:33.572525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:33.572724 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:33.573453 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:33.573481 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:33.575857 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:33.577907 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:33.577923 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:33.577942 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:33.578300 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:33.578634 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:33.578638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:33.578640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:33.578867 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:33.579342 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:33.579348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:33.579767 (+ 419us) client_negotiation.cc:769] Sending connection context
0924 04:05:33.580591 (+ 824us) client_negotiation.cc:240] Negotiation successful
0924 04:05:33.581032 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":156,"threads_started":1}
I20260924 04:05:33.583518 13144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36933
I20260924 04:05:33.583777 13144 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:33.584383 13144 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
I20260924 04:05:33.585556 12660 ts_manager.cc:194] Registered new tserver with Master: ea9791a31a734daaa435b46117af4c88 (127.0.98.3:40981)
I20260924 04:05:33.586321 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51737
I20260924 04:05:33.592291 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:05:33.593127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 12741
I20260924 04:05:33.599771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 12880
I20260924 04:05:33.606148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 13015
I20260924 04:05:33.612285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 12628
I20260924 04:05:33.617612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36933
--webserver_interface=127.0.98.62
--webserver_port=40991
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36933
--ranger_config_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33167/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33167/kms with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:33.727802 13147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:33.728128 13147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:33.728171 13147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:33.732435 13147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:33.732523 13147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:33.732543 13147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:33.732559 13147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:33.732573 13147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:33.738523 13147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33167/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36933
--ranger_config_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:33.739660 13147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:33.740731 13147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:33.749477 13153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:33.749454 13152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:33.749563 13147 server_base.cc:1034] running on GCE node
W20260924 04:05:33.749460 13155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:33.750257 13147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:33.751381 13147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:33.752537 13147 hybrid_clock.cc:648] HybridClock initialized: now 1790222733752517 us; error 31 us; skew 500 ppm
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:33.755119 13147 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:05:33.756001 13147 webserver.cc:460] Webserver started at http://127.0.98.62:40991/ using document root <none> and password file <none>
I20260924 04:05:33.756523 13147 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:33.756597 13147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:05:33 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:33.822165 13147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:33.824586 13163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:33.826048 13147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:33.826179 13147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "6306ce75e76c4e0fb97dfba94671ecf8"
format_stamp: "Formatted at 2026-09-24 04:05:31 on dist-test-slave-zcsk"
server_key: "f928b00afc349f8dccd15dfe5b889589"
server_key_iv: "27b11610a7f14fb266f531ed79db9ab1"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:33.826778 13147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:33.838374 13147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:33.841812 13147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:33.842006 13147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:33.849341 13147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36933
I20260924 04:05:33.849355 13215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36933 every 8 connection(s)
I20260924 04:05:33.850450 13147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:05:33.855144 13216 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:05:33.855778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 13147
I20260924 04:05:33.856314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44669
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36933
--webserver_port=42397
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33167/kms with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:33.857515 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Bootstrap starting.
I20260924 04:05:33.874598 13216 log.cc:824] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:33.881383 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:05:33.881762 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Bootstrap complete.
I20260924 04:05:33.885708 13216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } }
I20260924 04:05:33.886121 13216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6306ce75e76c4e0fb97dfba94671ecf8, State: Initialized, Role: FOLLOWER
I20260924 04:05:33.886650 13216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } }
I20260924 04:05:33.886802 13216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:05:33.886883 13216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:05:33.886981 13216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:05:33.889058 13216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } }
I20260924 04:05:33.889387 13216 leader_election.cc:302] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6306ce75e76c4e0fb97dfba94671ecf8; no voters:
I20260924 04:05:33.889674 13216 leader_election.cc:288] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:05:33.889776 13220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:05:33.889971 13220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [term 2 LEADER]: Becoming Leader. State: Replica: 6306ce75e76c4e0fb97dfba94671ecf8, State: Running, Role: LEADER
I20260924 04:05:33.890338 13220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } }
I20260924 04:05:33.890969 13216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:05:33.891273 13220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6306ce75e76c4e0fb97dfba94671ecf8. Latest consensus state: current_term: 2 leader_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } } }
I20260924 04:05:33.891943 13220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:33.891803 13221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6306ce75e76c4e0fb97dfba94671ecf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36933 } } }
I20260924 04:05:33.892593 13221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:33.892665 13229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:05:33.895838 13229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:05:33.896415 13216 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:05:33.896600 13229 catalog_manager.cc:1259] Loaded cluster ID: 82971a98b03d4f16977eb12b663fc706
I20260924 04:05:33.896698 13229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:05:33.900014 13229 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:05:33.900729 13229 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Loaded TSK: 0
I20260924 04:05:33.902195 13229 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:05:33.993204 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:33.993475 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:33.993537 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:33.997710 13218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:33.997778 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:33.997859 13218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:05:34.004140 13218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42397
--tserver_master_addrs=127.0.98.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:34.005769 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:34.007223 13218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:34.022166 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:34.027372 13254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:34.027948 13256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:34.029639 13218 server_base.cc:1034] running on GCE node
I20260924 04:05:34.030227 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:34.031153 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:34.032526 13218 hybrid_clock.cc:648] HybridClock initialized: now 1790222734032477 us; error 66 us; skew 500 ppm
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:34.036402 13218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:05:34.037734 13218 webserver.cc:460] Webserver started at http://127.0.98.1:42397/ using document root <none> and password file <none>
I20260924 04:05:34.038569 13218 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:34.038678 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:34.062600 13218 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.004s
I20260924 04:05:34.065831 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.067458 13218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:05:34.067584 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "dd0a9b3bb416458189c3668470f469c4"
format_stamp: "Formatted at 2026-09-24 04:05:32 on dist-test-slave-zcsk"
server_key: "0a6fd10db569dde34fd799b44dc0e956"
server_key_iv: "1a546302124ac8bec0ddf2f0437c063f"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:34.068351 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:34.082165 13218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:34.085860 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:34.086134 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:34.086911 13218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:34.088066 13218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:34.088127 13218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.088191 13218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:34.088222 13218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.098731 13218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44669
I20260924 04:05:34.098907 13377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44669 every 8 connection(s)
I20260924 04:05:34.099834 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:05:34.105405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 13218
I20260924 04:05:34.105917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42333
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36933
--webserver_port=35645
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33167/kms with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:34.120651 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:34.102513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36933 (local address 127.0.98.1:34037)
0924 04:05:34.103226 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:34.103280 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:34.104237 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:34.105436 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:34.105444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:34.105849 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:34.106417 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:34.106433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.108042 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.108046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:34.109369 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:34.109381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.110480 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.110485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:34.110712 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:34.111476 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:34.111498 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:34.113437 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:34.116150 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:34.116157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:34.116173 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:34.116556 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:34.117005 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:34.117008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:34.117010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:34.117191 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:34.118273 (+ 1082us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:34.118280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:34.118766 (+ 486us) client_negotiation.cc:769] Sending connection context
0924 04:05:34.119014 (+ 248us) client_negotiation.cc:240] Negotiation successful
0924 04:05:34.119428 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":137,"threads_started":1}
I20260924 04:05:34.123131 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:34.102316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34037 (local address 127.0.98.62:36933)
0924 04:05:34.102945 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:34.102968 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:34.104001 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 04:05:34.104680 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:34.104690 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:34.105034 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:34.105338 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:34.106586 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.107859 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.109527 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.110274 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.111141 (+ 867us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:34.113727 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:34.113734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:34.113754 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:34.113791 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:34.115990 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:34.116714 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:34.116720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:34.116726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:34.116880 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:34.117322 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:34.117325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:34.117327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:34.118091 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:34.121706 (+ 3615us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:34.121865 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 04:05:34.122161 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":106,"threads_started":1}
I20260924 04:05:34.126582 13378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36933
I20260924 04:05:34.126860 13378 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.127427 13378 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
W20260924 04:05:34.129839 13378 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36933 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:05:34.129973 13378 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.130151 13378 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
I20260924 04:05:34.131206 13378 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.131431 13378 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
W20260924 04:05:34.132318 13378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:05:34.254062 13382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:34.254379 13382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:34.254448 13382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:34.258868 13382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:34.258944 13382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:34.259029 13382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:05:34.265014 13382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35645
--tserver_master_addrs=127.0.98.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:34.266403 13382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:34.267551 13382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:34.277860 13390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:34.277858 13387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:34.277858 13388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:34.278286 13382 server_base.cc:1034] running on GCE node
I20260924 04:05:34.278690 13382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:34.279320 13382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:34.280500 13382 hybrid_clock.cc:648] HybridClock initialized: now 1790222734280470 us; error 46 us; skew 500 ppm
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:34.283390 13382 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:05:34.284628 13382 webserver.cc:460] Webserver started at http://127.0.98.2:35645/ using document root <none> and password file <none>
I20260924 04:05:34.285388 13382 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:34.285497 13382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:34.311030 13382 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260924 04:05:34.315770 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.317497 13382 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 04:05:34.317644 13382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "03cb21bee90c42c78f51dda72f8f856c"
format_stamp: "Formatted at 2026-09-24 04:05:33 on dist-test-slave-zcsk"
server_key: "7e2192cc54d471e1f74cfabafd29a42a"
server_key_iv: "aa9872ef76bbe0bc29e381edb31c9749"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:34.318423 13382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:34.345582 13382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:34.349488 13382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:34.349695 13382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:34.350417 13382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:34.351469 13382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:34.351528 13382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.351579 13382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:34.351600 13382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.362121 13382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42333
I20260924 04:05:34.363533 13382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:05:34.365949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 13382
I20260924 04:05:34.366444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33167/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40981
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36933
--webserver_port=33639
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33167/kms with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:34.369578 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42333 every 8 connection(s)
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:34.395970 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:34.372166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36933 (local address 127.0.98.2:46935)
0924 04:05:34.380398 (+ 8232us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:34.380502 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:34.382036 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:34.382981 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:34.382991 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:34.383377 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:34.384164 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:34.384186 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.385384 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.385388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:34.386879 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:34.386890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.387769 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.387773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:34.388024 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:34.388721 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:34.388743 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:34.390566 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:34.392622 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:34.392629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:34.392647 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:34.393017 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:34.393445 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:34.393448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:34.393450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:34.393625 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:34.394102 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:34.394108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:34.394423 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 04:05:34.394614 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 04:05:34.394962 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7903,"thread_start_us":175,"threads_started":1}
I20260924 04:05:34.396606 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:34.371511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46935 (local address 127.0.98.62:36933)
0924 04:05:34.371671 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:34.371676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:34.380647 (+ 8971us) server_negotiation.cc:407] Connection header received
0924 04:05:34.382377 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:34.382382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:34.382444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:34.382509 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:34.384349 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.385220 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.387044 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.387475 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.387555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:34.390776 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:34.390780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:34.390784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:34.390819 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:34.392481 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:34.393165 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:34.393169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:34.393171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:34.393256 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:34.393746 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:34.393749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:34.393751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:34.393968 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:34.394052 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:34.396333 (+ 2281us) server_negotiation.cc:299] Negotiation successful
0924 04:05:34.396464 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 04:05:34.398115 13512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36933
I20260924 04:05:34.398365 13512 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.398931 13512 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
W20260924 04:05:34.400549 13512 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36933 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:05:34.400671 13512 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.400852 13512 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
I20260924 04:05:34.401701 13512 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.401877 13512 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
W20260924 04:05:34.402572 13512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:05:34.504807 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:34.505079 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:34.505139 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:34.509378 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:34.509454 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:34.509537 13514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:05:34.515519 13514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33167/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33639
--tserver_master_addrs=127.0.98.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:34.516705 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:34.517771 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:34.528661 13524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:34.529124 13514 server_base.cc:1034] running on GCE node
W20260924 04:05:34.532095 13521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:34.528718 13522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:34.540313 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:34.541298 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:34.542511 13514 hybrid_clock.cc:648] HybridClock initialized: now 1790222734542475 us; error 63 us; skew 500 ppm
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:34.546180 13514 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:05:34.547230 13514 webserver.cc:460] Webserver started at http://127.0.98.3:33639/ using document root <none> and password file <none>
I20260924 04:05:34.547804 13514 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:34.547896 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:34.566035 13514 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:05:34.568632 13532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.569914 13514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:34.570030 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "ea9791a31a734daaa435b46117af4c88"
format_stamp: "Formatted at 2026-09-24 04:05:33 on dist-test-slave-zcsk"
server_key: "a8d47dd1ba4ac6b696601d6ed23911e6"
server_key_iv: "a3ec836426b960670a27b14ae2f80356"
server_key_version: "kuduclusterkey@0"
I20260924 04:05:34.570590 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:34.597873 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:34.601054 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:34.601246 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:34.601883 13514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:34.602869 13514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:34.602988 13514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.603087 13514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:34.603152 13514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:34.614405 13514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40981
I20260924 04:05:34.614459 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40981 every 8 connection(s)
I20260924 04:05:34.615892 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:05:34.625152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 13514
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:34.635092 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:34.619700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44575 (local address 127.0.98.62:36933)
0924 04:05:34.619838 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:34.619843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:34.620321 (+ 478us) server_negotiation.cc:407] Connection header received
0924 04:05:34.621606 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:34.621610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:34.621667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:34.621719 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:34.623863 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.624729 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.626934 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.627271 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.627355 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:34.630538 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:34.630542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:34.630545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:34.630573 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:34.632325 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:34.633051 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:34.633055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:34.633057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:34.633153 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:34.633636 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:34.633640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:34.633642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:34.633883 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:34.633960 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:34.634781 (+ 821us) server_negotiation.cc:299] Negotiation successful
0924 04:05:34.634918 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:05:34.637223 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:34.618522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36933 (local address 127.0.98.3:44575)
0924 04:05:34.620153 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:34.620210 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:34.621339 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:34.622197 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:34.622206 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:34.622647 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:34.623669 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:34.623692 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.624905 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.624908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:34.626781 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:34.626793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.627566 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.627570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:34.627791 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:34.628490 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:34.628512 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:34.630331 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:34.632494 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:34.632501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:34.632523 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:34.632921 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:34.633271 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:34.633274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:34.633277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:34.633513 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:34.634046 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:34.634053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:34.634463 (+ 410us) client_negotiation.cc:769] Sending connection context
0924 04:05:34.635584 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0924 04:05:34.636021 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1398,"thread_start_us":151,"threads_started":1}
I20260924 04:05:34.638924 13646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36933
I20260924 04:05:34.639211 13646 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.639848 13646 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
W20260924 04:05:34.641661 13646 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36933 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:05:34.641790 13646 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.641957 13646 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
I20260924 04:05:34.642865 13646 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:34.642953 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:34.625890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.0.98.62:36933)
0924 04:05:34.628275 (+ 2385us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:34.628318 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:34.629576 (+ 1258us) server_negotiation.cc:407] Connection header received
0924 04:05:34.629620 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:34.629623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:34.629672 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:34.629746 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:34.631480 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.632386 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.633641 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:34.634010 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:34.634103 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:34.635757 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:34.635761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:34.635764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:34.635787 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:34.637489 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:34.640740 (+ 3251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:34.640743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:34.640745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:34.640857 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:34.641177 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:34.641180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:34.641182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:34.641391 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:34.641594 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:34.642651 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0924 04:05:34.642784 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2304,"thread_start_us":107,"threads_started":1}
I20260924 04:05:34.643057 13646 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
W20260924 04:05:34.643846 13646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:05:34 dist-test-slave-zcsk krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:35.132530 13378 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:35.132810 13378 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
I20260924 04:05:35.133879 13178 ts_manager.cc:194] Registered new tserver with Master: dd0a9b3bb416458189c3668470f469c4 (127.0.98.1:44669)
I20260924 04:05:35.135462 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34037
I20260924 04:05:35.402818 13512 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:35.403277 13512 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
I20260924 04:05:35.404202 13178 ts_manager.cc:194] Registered new tserver with Master: 03cb21bee90c42c78f51dda72f8f856c (127.0.98.2:42333)
I20260924 04:05:35.405022 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46935
Sep 24 04:05:35 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:35.644037 13646 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:35.644364 13646 heartbeater.cc:511] Master 127.0.98.62:36933 requested a full tablet report, sending...
Sep 24 04:05:35 dist-test-slave-zcsk krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:05:35.652064 13178 ts_manager.cc:194] Registered new tserver with Master: ea9791a31a734daaa435b46117af4c88 (127.0.98.3:40981)
I20260924 04:05:35.652660 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44575
I20260924 04:05:35.663157 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:05:35.683750 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:35.667998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.0.98.62:36933)
0924 04:05:35.668481 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:35.668486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:35.671677 (+ 3191us) server_negotiation.cc:407] Connection header received
0924 04:05:35.671974 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:35.671977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:35.672039 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:35.672117 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:35.675325 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.676282 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.677504 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.677866 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.677961 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:35.679682 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:35.679686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:35.679689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:35.679721 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:35.681506 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:35.681993 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:35.681997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:35.681999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:35.682094 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:35.682397 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:35.682401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:35.682402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:35.682644 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:35.682758 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:35.683422 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 04:05:35.683555 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":126,"threads_started":1}
I20260924 04:05:35.688544 13178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:05:35.693540 13178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:05:35.714673 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:35.707143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47792 (local address 127.0.98.2:42333)
0924 04:05:35.708709 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:35.708716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:35.708951 (+ 235us) server_negotiation.cc:407] Connection header received
0924 04:05:35.709046 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:35.709052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:35.709303 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:35.709449 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:35.709767 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.710985 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.712837 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.713722 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.713804 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:35.713951 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:35.714407 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 04:05:35.714515 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1459,"thread_start_us":109,"threads_started":1}
I20260924 04:05:35.715453 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:35.706753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42333 (local address 127.0.0.1:47792)
0924 04:05:35.708834 (+ 2081us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:35.708846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:35.708936 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:35.709473 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:35.709479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:35.709505 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:35.709646 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:35.709653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.711115 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.711118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:35.712694 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:35.712705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.714052 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.714056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:35.714183 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:35.714229 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:05:35.715149 (+ 920us) client_negotiation.cc:240] Negotiation successful
0924 04:05:35.715292 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2026,"thread_start_us":81,"threads_started":1}
I20260924 04:05:35.717655 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:35.707976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44669 (local address 127.0.0.1:50088)
0924 04:05:35.712110 (+ 4134us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:35.712125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:35.712225 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:35.712929 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:35.712932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:35.712953 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:35.713049 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:35.713055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.714644 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.714648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:35.715870 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:35.715881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.717248 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.717251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:35.717382 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:35.717406 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:05:35.717464 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:05:35.717527 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4059,"mutex_wait_us":22,"thread_start_us":94,"threads_started":1}
I20260924 04:05:35.719261 13446 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfc82a6ca244ee2bf6f7b897600521a (DEFAULT_TABLE table=test-table [id=db81cd7516df4e458ff9e975ab48326b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:35.720359 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:35.705572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40981 (local address 127.0.0.1:35406)
0924 04:05:35.708231 (+ 2659us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:35.708289 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:35.708431 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:35.712846 (+ 4415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:35.712849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:35.712868 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:35.712957 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:35.712962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.717038 (+ 4076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.717042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:35.718894 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:35.718905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.719979 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.719984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:35.720077 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:35.720101 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:05:35.720147 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 04:05:35.720226 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2466,"mutex_wait_us":729,"thread_start_us":105,"threads_started":1}
I20260924 04:05:35.721240 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:35.705697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.0.98.3:40981)
0924 04:05:35.712142 (+ 6445us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:35.712149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:35.712176 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:05:35.712255 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:35.712262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:35.712506 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:35.712669 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:35.713083 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.714124 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.719040 (+ 4916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.719832 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.720786 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:35.720946 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:35.720998 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 04:05:35.721093 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6321,"thread_start_us":135,"threads_started":1}
I20260924 04:05:35.721414 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfc82a6ca244ee2bf6f7b897600521a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:35.721942 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:35.708392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.0.98.1:44669)
0924 04:05:35.712476 (+ 4084us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:35.712483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:35.712506 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:05:35.712578 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:35.712584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:35.712781 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:35.713219 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:35.713395 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.714514 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.716058 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:35.716913 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:35.721484 (+ 4571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:35.721651 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:35.721706 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 04:05:35.721804 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3971,"thread_start_us":120,"threads_started":1}
I20260924 04:05:35.725416 13579 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfc82a6ca244ee2bf6f7b897600521a (DEFAULT_TABLE table=test-table [id=db81cd7516df4e458ff9e975ab48326b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:35.726606 13579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfc82a6ca244ee2bf6f7b897600521a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:35.737677 13312 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfc82a6ca244ee2bf6f7b897600521a (DEFAULT_TABLE table=test-table [id=db81cd7516df4e458ff9e975ab48326b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:35.738561 13682 tablet_bootstrap.cc:492] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: Bootstrap starting.
I20260924 04:05:35.739001 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfc82a6ca244ee2bf6f7b897600521a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:35.741166 13682 tablet_bootstrap.cc:654] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:35.741658 13683 tablet_bootstrap.cc:492] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Bootstrap starting.
I20260924 04:05:35.742674 13682 log.cc:824] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:35.745185 13683 tablet_bootstrap.cc:654] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:35.746541 13683 log.cc:824] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:35.749032 13682 tablet_bootstrap.cc:492] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: No bootstrap required, opened a new log
I20260924 04:05:35.749541 13682 ts_tablet_manager.cc:1397] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260924 04:05:35.755151 13686 tablet_bootstrap.cc:492] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4: Bootstrap starting.
I20260924 04:05:35.758559 13686 tablet_bootstrap.cc:654] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:35.759661 13686 log.cc:824] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:35.762571 13682 raft_consensus.cc:348] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:35.766554 13682 raft_consensus.cc:374] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:35.766857 13682 raft_consensus.cc:729] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03cb21bee90c42c78f51dda72f8f856c, State: Initialized, Role: FOLLOWER
I20260924 04:05:35.767625 13682 consensus_queue.cc:260] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:35.771055 13686 tablet_bootstrap.cc:492] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4: No bootstrap required, opened a new log
I20260924 04:05:35.771296 13686 ts_tablet_manager.cc:1397] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260924 04:05:35.774331 13682 ts_tablet_manager.cc:1428] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260924 04:05:35.776194 13686 raft_consensus.cc:348] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:35.776648 13686 raft_consensus.cc:374] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:35.776721 13686 raft_consensus.cc:729] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0a9b3bb416458189c3668470f469c4, State: Initialized, Role: FOLLOWER
I20260924 04:05:35.776803 13683 tablet_bootstrap.cc:492] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: No bootstrap required, opened a new log
I20260924 04:05:35.777020 13683 ts_tablet_manager.cc:1397] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20260924 04:05:35.777284 13686 consensus_queue.cc:260] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:35.778087 13378 heartbeater.cc:503] Master 127.0.98.62:36933 was elected leader, sending a full tablet report...
I20260924 04:05:35.778532 13686 ts_tablet_manager.cc:1428] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 04:05:35.789621 13512 heartbeater.cc:503] Master 127.0.98.62:36933 was elected leader, sending a full tablet report...
I20260924 04:05:35.788702 13683 raft_consensus.cc:348] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:35.792781 13683 raft_consensus.cc:374] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:35.792858 13683 raft_consensus.cc:729] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9791a31a734daaa435b46117af4c88, State: Initialized, Role: FOLLOWER
I20260924 04:05:35.793848 13683 consensus_queue.cc:260] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:35.795156 13683 ts_tablet_manager.cc:1428] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260924 04:05:35.795547 13646 heartbeater.cc:503] Master 127.0.98.62:36933 was elected leader, sending a full tablet report...
W20260924 04:05:35.853065 13379 tablet.cc:2369] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:05:35.865734 13513 tablet.cc:2369] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:05:35.868142 13647 tablet.cc:2369] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:05:36.000172 13688 raft_consensus.cc:482] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:36.000416 13688 raft_consensus.cc:504] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:36.001838 13688 leader_election.cc:288] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea9791a31a734daaa435b46117af4c88 (127.0.98.3:40981), dd0a9b3bb416458189c3668470f469c4 (127.0.98.1:44669)
I20260924 04:05:36.006561 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:36.002062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40981 (local address 127.0.98.2:37675)
0924 04:05:36.002590 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:36.002601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:36.002693 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:36.002997 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:36.002999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:36.003012 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:36.003102 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:36.003108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.003853 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.003856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:36.005464 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:36.005471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.006084 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.006087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:36.006193 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:36.006226 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:05:36.006312 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:05:36.006384 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":75,"thread_start_us":80,"threads_started":1}
I20260924 04:05:36.006564 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:36.002438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37675 (local address 127.0.98.3:40981)
0924 04:05:36.002561 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:36.002566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:36.002667 (+ 101us) server_negotiation.cc:407] Connection header received
0924 04:05:36.002788 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:36.002792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:36.002835 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:36.002893 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:36.003238 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.003748 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.005575 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.005977 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.006070 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:36.006135 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:36.006350 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 04:05:36.006436 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:05:36.006608 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:36.002251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36039 (local address 127.0.98.1:44669)
0924 04:05:36.002427 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:36.002431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:36.002512 (+ 81us) server_negotiation.cc:407] Connection header received
0924 04:05:36.002746 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:36.002750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:36.002806 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:36.002888 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:36.003187 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.004017 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.005376 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.006013 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.006115 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:36.006177 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:36.006424 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 04:05:36.006506 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 04:05:36.007216 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfc82a6ca244ee2bf6f7b897600521a" candidate_uuid: "03cb21bee90c42c78f51dda72f8f856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9791a31a734daaa435b46117af4c88" is_pre_election: true
I20260924 04:05:36.007373 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:36.002060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44669 (local address 127.0.98.2:36039)
0924 04:05:36.002432 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:36.002446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:36.002589 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:36.002915 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:36.002918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:36.002933 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:36.003071 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:36.003078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.004127 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.004130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:36.005235 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:36.005246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.006201 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.006205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:36.006300 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:36.006320 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:05:36.007172 (+ 852us) client_negotiation.cc:240] Negotiation successful
0924 04:05:36.007248 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":128,"threads_started":1}
I20260924 04:05:36.007592 13598 raft_consensus.cc:2454] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03cb21bee90c42c78f51dda72f8f856c in term 0.
I20260924 04:05:36.008004 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfc82a6ca244ee2bf6f7b897600521a" candidate_uuid: "03cb21bee90c42c78f51dda72f8f856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0a9b3bb416458189c3668470f469c4" is_pre_election: true
I20260924 04:05:36.008102 13400 leader_election.cc:302] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03cb21bee90c42c78f51dda72f8f856c, ea9791a31a734daaa435b46117af4c88; no voters:
I20260924 04:05:36.008304 13332 raft_consensus.cc:2454] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03cb21bee90c42c78f51dda72f8f856c in term 0.
I20260924 04:05:36.008339 13688 raft_consensus.cc:2781] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:05:36.008420 13688 raft_consensus.cc:482] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:05:36.008457 13688 raft_consensus.cc:3037] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:36.009495 13688 raft_consensus.cc:504] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:36.009840 13688 leader_election.cc:288] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [CANDIDATE]: Term 1 election: Requested vote from peers ea9791a31a734daaa435b46117af4c88 (127.0.98.3:40981), dd0a9b3bb416458189c3668470f469c4 (127.0.98.1:44669)
I20260924 04:05:36.010187 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfc82a6ca244ee2bf6f7b897600521a" candidate_uuid: "03cb21bee90c42c78f51dda72f8f856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9791a31a734daaa435b46117af4c88"
I20260924 04:05:36.010289 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfc82a6ca244ee2bf6f7b897600521a" candidate_uuid: "03cb21bee90c42c78f51dda72f8f856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0a9b3bb416458189c3668470f469c4"
I20260924 04:05:36.010320 13598 raft_consensus.cc:3037] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:36.010409 13332 raft_consensus.cc:3037] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:36.011597 13598 raft_consensus.cc:2454] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03cb21bee90c42c78f51dda72f8f856c in term 1.
I20260924 04:05:36.011597 13332 raft_consensus.cc:2454] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03cb21bee90c42c78f51dda72f8f856c in term 1.
I20260924 04:05:36.011943 13399 leader_election.cc:302] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03cb21bee90c42c78f51dda72f8f856c, dd0a9b3bb416458189c3668470f469c4; no voters:
I20260924 04:05:36.012178 13688 raft_consensus.cc:2781] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:05:36.012406 13688 raft_consensus.cc:686] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 1 LEADER]: Becoming Leader. State: Replica: 03cb21bee90c42c78f51dda72f8f856c, State: Running, Role: LEADER
I20260924 04:05:36.016278 13688 consensus_queue.cc:237] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } }
I20260924 04:05:36.020363 13178 catalog_manager.cc:5495] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c reported cstate change: term changed from 0 to 1, leader changed from <none> to 03cb21bee90c42c78f51dda72f8f856c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "03cb21bee90c42c78f51dda72f8f856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03cb21bee90c42c78f51dda72f8f856c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 } health_report { overall_health: UNKNOWN } } }
I20260924 04:05:36.081446 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:36.077408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.0.98.2:42333)
0924 04:05:36.077543 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:36.077547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:36.077563 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:05:36.077687 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:36.077690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:36.077759 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:36.077863 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:05:36.078073 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.078854 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.079675 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:36.080058 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:36.080158 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:36.080469 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:05:36.081025 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:05:36.081146 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:36.081185 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 04:05:36.081248 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:05:36.087990 13332 raft_consensus.cc:1261] T 5bfc82a6ca244ee2bf6f7b897600521a P dd0a9b3bb416458189c3668470f469c4 [term 1 FOLLOWER]: Refusing update from remote peer 03cb21bee90c42c78f51dda72f8f856c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:05:36.087986 13598 raft_consensus.cc:1261] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 1 FOLLOWER]: Refusing update from remote peer 03cb21bee90c42c78f51dda72f8f856c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:05:36.088673 13688 consensus_queue.cc:1035] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9791a31a734daaa435b46117af4c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:05:36.089069 13700 consensus_queue.cc:1035] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0a9b3bb416458189c3668470f469c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:05:36.097891 13703 mvcc.cc:205] Tried to move back new op lower bound from 7332752327009185792 to 7332752326726496256. Current Snapshot: MvccSnapshot[applied={T|T < 7332752327009185792}]
I20260924 04:05:36.108008 13180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:05:36.108201 13180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:05:36.110751 13180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6306ce75e76c4e0fb97dfba94671ecf8: Sending DeleteTablet for 3 replicas of tablet 5bfc82a6ca244ee2bf6f7b897600521a
I20260924 04:05:36.111526 13446 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfc82a6ca244ee2bf6f7b897600521a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47792
I20260924 04:05:36.112020 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfc82a6ca244ee2bf6f7b897600521a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50088
I20260924 04:05:36.112061 13579 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfc82a6ca244ee2bf6f7b897600521a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35406
I20260924 04:05:36.113329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 13218
I20260924 04:05:36.113474 13704 mvcc.cc:205] Tried to move back new op lower bound from 7332752327009185792 to 7332752326726496256. Current Snapshot: MvccSnapshot[applied={T|T < 7332752327009185792 or (T in {7332752327009185792})}]
I20260924 04:05:36.114078 13713 tablet_replica.cc:330] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: stopping tablet replica
I20260924 04:05:36.114352 13715 tablet_replica.cc:330] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: stopping tablet replica
I20260924 04:05:36.114753 13715 raft_consensus.cc:2229] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:05:36.115080 13713 raft_consensus.cc:2229] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:05:36.115491 13713 raft_consensus.cc:2258] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:05:36.115813 13715 raft_consensus.cc:2258] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:05:36.117668 13713 ts_tablet_manager.cc:1905] T 5bfc82a6ca244ee2bf6f7b897600521a P 03cb21bee90c42c78f51dda72f8f856c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:05:36.122520 13164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44669 (error 108)
I20260924 04:05:36.123355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 13382
I20260924 04:05:36.124167 13715 ts_tablet_manager.cc:1905] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:05:36.126786 13715 ts_tablet_manager.cc:1918] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:05:36.126921 13715 log.cc:1196] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/5bfc82a6ca244ee2bf6f7b897600521a
I20260924 04:05:36.127259 13715 ts_tablet_manager.cc:1939] T 5bfc82a6ca244ee2bf6f7b897600521a P ea9791a31a734daaa435b46117af4c88: Deleting consensus metadata
I20260924 04:05:36.128603 13165 catalog_manager.cc:4841] TS ea9791a31a734daaa435b46117af4c88 (127.0.98.3:40981): tablet 5bfc82a6ca244ee2bf6f7b897600521a (table test-table [id=db81cd7516df4e458ff9e975ab48326b]) successfully deleted
I20260924 04:05:36.129024 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:36.128454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44669 (local address 127.0.0.1:50102)
0924 04:05:36.128871 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:36.128940 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338}
W20260924 04:05:36.129185 13164 catalog_manager.cc:4584] TS dd0a9b3bb416458189c3668470f469c4 (127.0.98.1:44669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bfc82a6ca244ee2bf6f7b897600521a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44669: connect: Connection refused (error 111)
I20260924 04:05:36.131433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 13514
I20260924 04:05:36.133227 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:36.131686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42333 (local address 127.0.0.1:47798)
0924 04:05:36.133071 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:36.133133 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1293}
I20260924 04:05:36.137209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 13147
I20260924 04:05:36.143431 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:05:36.478628 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:05:36.478739 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:05:36.813578 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:05:36.813 UTC [12214] LOG: received smart shutdown request
2026-09-24 04:05:36.816 UTC [12214] LOG: background worker "logical replication launcher" (PID 12223) exited with exit code 1
2026-09-24 04:05:36.817 UTC [12218] LOG: shutting down
2026-09-24 04:05:36.993 UTC [12214] LOG: database system is shut down
2026-09-24T04:05:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49395 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:05:37 dist-test-slave-zcsk krb5kdc[13725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:05:37 dist-test-slave-zcsk krb5kdc[13725](info): set up 2 sockets
Sep 24 04:05:37 dist-test-slave-zcsk krb5kdc[13725](info): commencing operation
krb5kdc: starting...
W20260924 04:05:38.043780 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:05:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:05:38.163208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34363
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34363,127.0.98.61:33269,127.0.98.60:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:38.272817 13738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:38.273145 13738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:38.273204 13738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:38.277441 13738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:38.277522 13738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:38.277549 13738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:38.277571 13738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:38.277601 13738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:38.283170 13738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34363,127.0.98.61:33269,127.0.98.60:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:38.284384 13738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:38.285410 13738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:38.293864 13743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:38.293846 13744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:38.293864 13746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:38.294049 13738 server_base.cc:1034] running on GCE node
I20260924 04:05:38.294770 13738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:38.295890 13738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:38.297092 13738 hybrid_clock.cc:648] HybridClock initialized: now 1790222738297061 us; error 39 us; skew 500 ppm
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:38.299841 13738 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:05:38.300745 13738 webserver.cc:460] Webserver started at http://127.0.98.62:33307/ using document root <none> and password file <none>
I20260924 04:05:38.301271 13738 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:38.301334 13738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:38.301582 13738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:05:38.303650 13738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "184cb5ac53bc4d6dac884e02424dd5e1"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "0901d2e37ffd2a9c0c1056d298b0bb34"
server_key_iv: "8774d0af0fb7d89f2021fe86db889373"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.304205 13738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "184cb5ac53bc4d6dac884e02424dd5e1"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "0901d2e37ffd2a9c0c1056d298b0bb34"
server_key_iv: "8774d0af0fb7d89f2021fe86db889373"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.307950 13738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:05:38.310413 13753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:38.311771 13738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:05:38.311899 13738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "184cb5ac53bc4d6dac884e02424dd5e1"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "0901d2e37ffd2a9c0c1056d298b0bb34"
server_key_iv: "8774d0af0fb7d89f2021fe86db889373"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.312036 13738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:38.323062 13738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:38.325930 13738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:38.326102 13738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:38.332727 13738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34363
I20260924 04:05:38.332775 13805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34363 every 8 connection(s)
I20260924 04:05:38.333659 13738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:05:38.336505 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:38.339347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 13738
I20260924 04:05:38.339485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:05:38.339721 392 external_mini_cluster.cc:1428] Setting key 232bf8c955d700b6263a7cf8b29a911e
I20260924 04:05:38.340296 13806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:38.356297 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.342510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34363 (local address 127.0.0.1:60372)
0924 04:05:38.344134 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.344173 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.344967 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.345766 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.345770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.346120 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.346450 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.346464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.348072 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.348075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.348829 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.348838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.349452 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.349455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.349589 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.350172 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.350193 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.351677 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.353430 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.353438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.353448 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.353649 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.353933 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.353936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.353938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.354040 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.354594 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.354600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.354728 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.354874 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.355259 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"mutex_wait_us":150,"spinlock_wait_cycles":34688,"thread_start_us":129,"threads_started":1}
I20260924 04:05:38.356304 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.342510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.0.98.62:34363)
0924 04:05:38.342915 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.342932 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.344295 (+ 1363us) server_negotiation.cc:407] Connection header received
0924 04:05:38.345304 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.345310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.345573 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.345786 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.346591 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.347959 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.348936 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.349311 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.349473 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.351810 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.351815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.351826 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.351846 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.353329 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.353743 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.353748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.353749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.353847 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.354117 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.354120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.354122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.354488 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.354603 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.354955 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.355201 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260924 04:05:38.358920 13806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:05:38.359709 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.359394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56276)
0924 04:05:38.359559 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.359638 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 04:05:38.359966 13757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
I20260924 04:05:38.360790 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.360582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56286)
0924 04:05:38.360690 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.360735 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:05:38.360911 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:05:38.366812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33269
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34363,127.0.98.61:33269,127.0.98.60:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:38.410183 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } attempt: 1
I20260924 04:05:38.411019 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.410727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56294)
0924 04:05:38.410874 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.410941 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:05:38.411831 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.411574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56300)
0924 04:05:38.411707 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.411758 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 04:05:38.411986 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
W20260924 04:05:38.476967 13813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:38.477247 13813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:38.477288 13813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:05:38.480161 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } attempt: 2
I20260924 04:05:38.481086 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.480768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56314)
0924 04:05:38.480941 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.481010 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 04:05:38.481499 13813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:38.481567 13813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:38.481595 13813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:38.481627 13813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:38.481649 13813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:38.481846 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.481591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56316)
0924 04:05:38.481724 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.481778 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 04:05:38.481966 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33269: connect: Connection refused (error 111)
I20260924 04:05:38.487183 13813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34363,127.0.98.61:33269,127.0.98.60:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:38.488322 13813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:38.489333 13813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:38.497875 13819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:38.497905 13818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:38.497882 13821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:38.498371 13813 server_base.cc:1034] running on GCE node
I20260924 04:05:38.498942 13813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:38.500124 13813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:38.501276 13813 hybrid_clock.cc:648] HybridClock initialized: now 1790222738501256 us; error 36 us; skew 500 ppm
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:38.504074 13813 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:05:38.505019 13813 webserver.cc:460] Webserver started at http://127.0.98.61:35553/ using document root <none> and password file <none>
I20260924 04:05:38.505556 13813 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:38.505621 13813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:38.505818 13813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:05:38.507884 13813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/data/instance:
uuid: "3bbc620951a845db823f8b1375c45846"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "4cfed6a5518653ca9994e04251f6dbb8"
server_key_iv: "b61f00d7f11929d95269324749dd0a68"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.508479 13813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/wal/instance:
uuid: "3bbc620951a845db823f8b1375c45846"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "4cfed6a5518653ca9994e04251f6dbb8"
server_key_iv: "b61f00d7f11929d95269324749dd0a68"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.512214 13813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:05:38.514786 13828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:38.516199 13813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:38.516304 13813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/wal
uuid: "3bbc620951a845db823f8b1375c45846"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "4cfed6a5518653ca9994e04251f6dbb8"
server_key_iv: "b61f00d7f11929d95269324749dd0a68"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.516409 13813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:38.543107 13813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:38.546087 13813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:38.546245 13813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:38.553004 13813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33269
I20260924 04:05:38.553158 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33269 every 8 connection(s)
I20260924 04:05:38.553987 13813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:05:38.556869 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:38.560842 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:05:38.563563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 13813
I20260924 04:05:38.563701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-1/wal/instance
I20260924 04:05:38.564031 392 external_mini_cluster.cc:1428] Setting key 66d4fc8f7bac79e0b3beca687bdcf192
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 04:05:38.573127 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 04:05:38.576072 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.562908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.0.98.62:34363)
0924 04:05:38.563078 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.563082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.564104 (+ 1022us) server_negotiation.cc:407] Connection header received
0924 04:05:38.565176 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.565180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.565245 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.565343 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.566647 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.567196 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.568506 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.568773 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.568878 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.571634 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.571637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.571639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.571666 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.573050 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.573696 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.573699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.573701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.573788 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.574215 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.574219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.574221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.574459 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.574560 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.575753 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.575877 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:05:38.577414 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.563260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34363 (local address 127.0.0.1:60384)
0924 04:05:38.563941 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.563995 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.564924 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.565539 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.565547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.565924 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.566475 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.566490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.567335 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.567339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.568366 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.568377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.568934 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.568938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.569116 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.569813 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.569832 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.571441 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.573172 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.573179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.573198 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.573585 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.573897 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.573900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.573901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.574108 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.574616 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.574623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.574969 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.576008 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.576392 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":182,"threads_started":1}
I20260924 04:05:38.579020 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:05:38.583634 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.573623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56328)
0924 04:05:38.573829 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.573842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.573905 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.574535 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.574537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.574707 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.574809 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.574814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.576181 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.576185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.576986 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.576998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.577978 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.577981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.578088 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.578540 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.578554 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.580041 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.582050 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.582053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.582055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.582239 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.582500 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.582503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.582504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.582593 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.583129 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.583133 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.583238 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.583315 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.583459 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 04:05:38.584697 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.573753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.0.98.61:33269)
0924 04:05:38.574144 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.574156 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.574182 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:05:38.574223 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.574229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.574409 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.574538 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.574913 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.576046 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.577106 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.577874 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.577968 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.580182 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.580191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.580205 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.580226 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.581951 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.582336 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.582342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.582344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.582418 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.582677 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.582679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.582680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.582947 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.583950 (+ 1003us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.584076 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.584204 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":114,"threads_started":1}
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:05:38.586572 13806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:05:38.587277 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.587042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53890)
0924 04:05:38.587161 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.587214 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 04:05:38.588002 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.587717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53904)
0924 04:05:38.587858 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.587937 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 04:05:38.588171 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
I20260924 04:05:38.588359 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.579594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56340)
0924 04:05:38.579728 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.579743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.579824 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.580356 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.580359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.580505 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.580600 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.580605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.581624 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.581628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.582438 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.582448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.582940 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.582944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.583039 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.583397 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.583410 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.585246 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.587017 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.587020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.587022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.587187 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.587440 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.587442 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.587444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.587523 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.587937 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.587940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.588021 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.588103 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.588223 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:05:38.588750 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.579751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.0.98.61:33269)
0924 04:05:38.580131 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.580138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.580152 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:05:38.580187 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.580190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.580231 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.580316 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.580707 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.581503 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.582539 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.582826 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.582907 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.585406 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.585414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.585416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.585436 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.586916 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.587263 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.587266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.587268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.587342 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.587612 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.587615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.587616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.587827 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.587938 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.588540 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.588639 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"spinlock_wait_cycles":2944,"thread_start_us":182,"threads_started":1}
I20260924 04:05:38.589397 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:05:38.590000 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.589769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53920)
0924 04:05:38.589882 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.589951 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 04:05:38.590186 13832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:05:38.591008 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.590820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53936)
0924 04:05:38.590919 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.590963 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:05:38.591120 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
I20260924 04:05:38.591270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42055
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34363,127.0.98.61:33269,127.0.98.60:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:38.619344 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } attempt: 1
I20260924 04:05:38.620285 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.619950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53940)
0924 04:05:38.620111 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.620187 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:05:38.621065 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.620825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53944)
0924 04:05:38.620921 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.620973 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:05:38.621234 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
I20260924 04:05:38.640377 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } attempt: 1
I20260924 04:05:38.641319 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.640972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53956)
0924 04:05:38.641146 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.641220 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 04:05:38.642086 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.641848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53962)
0924 04:05:38.641959 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.642014 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:05:38.642251 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
I20260924 04:05:38.696400 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } attempt: 2
I20260924 04:05:38.697221 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.696918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53978)
0924 04:05:38.697061 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.697127 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 04:05:38.698060 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.697846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53980)
0924 04:05:38.697955 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.698000 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:05:38.698180 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
W20260924 04:05:38.702080 13889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:38.702335 13889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:38.702404 13889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:38.706784 13889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:38.706852 13889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:38.706879 13889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:38.706919 13889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:38.706950 13889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:38.710438 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } attempt: 2
I20260924 04:05:38.711256 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.710930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:53984)
0924 04:05:38.711096 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.711168 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:05:38.711954 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.711730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:54000)
0924 04:05:38.711818 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.711865 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 04:05:38.712069 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42055: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42055: connect: Connection refused (error 111)
I20260924 04:05:38.712812 13889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34363,127.0.98.61:33269,127.0.98.60:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:38.713881 13889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:38.714934 13889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:05:38.723739 13889 server_base.cc:1034] running on GCE node
W20260924 04:05:38.723701 13897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:38.723701 13895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:38.723709 13894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:38.724512 13889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:38.725633 13889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:38.726838 13889 hybrid_clock.cc:648] HybridClock initialized: now 1790222738726802 us; error 33 us; skew 500 ppm
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:38.729598 13889 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:05:38.730496 13889 webserver.cc:460] Webserver started at http://127.0.98.60:43257/ using document root <none> and password file <none>
I20260924 04:05:38.731058 13889 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:38.731129 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:38.731334 13889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:05:38.733417 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/data/instance:
uuid: "3d8fd3289c7a49dda4367c8257535803"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "c90158935c766d5c8315d5e5265d0e4f"
server_key_iv: "6982119098ad62f79c58c51ae663b699"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.733973 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/wal/instance:
uuid: "3d8fd3289c7a49dda4367c8257535803"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "c90158935c766d5c8315d5e5265d0e4f"
server_key_iv: "6982119098ad62f79c58c51ae663b699"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.737885 13889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:05:38.740584 13904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:38.741873 13889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:38.741999 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/wal
uuid: "3d8fd3289c7a49dda4367c8257535803"
format_stamp: "Formatted at 2026-09-24 04:05:38 on dist-test-slave-zcsk"
server_key: "c90158935c766d5c8315d5e5265d0e4f"
server_key_iv: "6982119098ad62f79c58c51ae663b699"
server_key_version: "encryptionkey@0"
I20260924 04:05:38.742106 13889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:38.773042 13889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:38.776468 13889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:38.776677 13889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:38.784267 13889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42055
I20260924 04:05:38.784296 13956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42055 every 8 connection(s)
I20260924 04:05:38.785724 13889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:05:38.787429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 13889
I20260924 04:05:38.787529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222576786845-392-0/minicluster-data/master-2/wal/instance
I20260924 04:05:38.787782 392 external_mini_cluster.cc:1428] Setting key e32b72b9765c4776a93fffcf0c772465
I20260924 04:05:38.789101 13957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:38.793965 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:05:38.797330 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } attempt: 3
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:05:38.803213 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } attempt: 3
I20260924 04:05:38.804659 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.789243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.0.98.60:42055)
0924 04:05:38.789747 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.789767 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.789813 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:05:38.790664 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.790689 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.791142 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.791543 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.792316 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.793778 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.794715 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.795362 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.795535 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.798367 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.798373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.798383 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.798409 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.800306 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.800717 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.800724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.800730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.800841 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.801147 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.801150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.801152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.801674 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.801895 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.802953 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.803277 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":184,"threads_started":1}
I20260924 04:05:38.806171 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.796141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.98.62:34363)
0924 04:05:38.796272 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.796276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.796547 (+ 271us) server_negotiation.cc:407] Connection header received
0924 04:05:38.796679 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.796682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.796728 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.796788 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.797458 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.798125 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.799196 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.799483 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.799588 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.802040 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.802043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.802045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.802069 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.803626 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.804560 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.804564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.804566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.804650 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.804972 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.804975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.804977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.805285 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.805437 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.805864 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.806020 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:05:38.806677 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.795970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34363 (local address 127.0.0.1:60392)
0924 04:05:38.796430 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.796470 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.796585 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.796851 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.796857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.797212 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.797337 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.797343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.798230 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.798233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.799089 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.799098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.799600 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.799604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.799786 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.800343 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.800358 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.801917 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.804126 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.804133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.804140 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.804440 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.804763 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.804766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.804767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.804848 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.805518 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.805525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.805647 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.806302 (+ 655us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.806495 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":59,"thread_start_us":163,"threads_started":1}
I20260924 04:05:38.807413 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.797739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:54014)
0924 04:05:38.797909 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.797921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.797985 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.798588 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.798591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.798741 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.798827 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.798835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.800112 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.800114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.800858 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.800870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.801428 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.801432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.801518 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.801946 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.801960 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.803303 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.805797 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.805799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.805801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.806094 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.806443 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.806446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.806448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.806533 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.806966 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.806969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.807059 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.807146 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.807268 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 04:05:38.807675 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:05:38.808457 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.797888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.0.98.60:42055)
0924 04:05:38.798361 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.798379 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.798391 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:05:38.798420 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.798423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.798461 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.798543 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.799218 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.799958 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.801016 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.801302 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.801400 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.803520 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.803524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.803527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.803551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.805685 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.806195 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.806198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.806200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.806343 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.806638 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.806640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.806642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.806832 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.807972 (+ 1140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.808043 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.808151 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":411,"thread_start_us":116,"threads_started":1}
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:05:38.812588 13806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1: Bootstrap starting.
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:05:38.814112 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.803628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:54024)
0924 04:05:38.803937 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.803952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.804034 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.804441 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.804444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.804615 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.804726 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.804734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.806769 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.806772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.807559 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.807571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.808388 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.808392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.808484 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.808943 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.808957 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.810348 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.812445 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.812448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.812450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.812696 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.812996 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.813000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.813002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.813108 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.813614 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.813617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.813716 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.813798 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.813939 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:38.814827 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.803881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.0.98.60:42055)
0924 04:05:38.804198 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.804206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.804218 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:05:38.804250 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.804253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.804301 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.805073 (+ 772us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.805192 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.805890 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.807956 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.808265 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.808705 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.810494 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.810498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.810501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.810524 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.812330 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.812818 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.812822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.812824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.812909 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.813222 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.813226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.813227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.813506 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.814415 (+ 909us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.814493 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.814604 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":131,"threads_started":1}
I20260924 04:05:38.816520 13806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:38.817381 13806 log.cc:824] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:38.818099 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846: Bootstrap starting.
I20260924 04:05:38.818915 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.808312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.0.98.61:33269)
0924 04:05:38.808422 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.808426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.809189 (+ 763us) server_negotiation.cc:407] Connection header received
0924 04:05:38.809332 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.809335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.809393 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.809444 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.809938 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.810741 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.811857 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.812224 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.812294 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.815074 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.815078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.815081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.815106 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.816975 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.817445 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.817449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.817451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.817537 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.817896 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.817900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.817901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.818176 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.818251 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.818655 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.818774 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:05:38.819579 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.808361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.0.98.62:34363)
0924 04:05:38.808510 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.808514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.808642 (+ 128us) server_negotiation.cc:407] Connection header received
0924 04:05:38.809075 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.809078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.809122 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.809187 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.809967 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.810733 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.811857 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.812222 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.812294 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.815480 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.815484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.815486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.815513 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.817685 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.818179 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.818183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.818184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.818266 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.818560 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.818564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.818565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.818762 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.818925 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.819292 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.819435 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 04:05:38.819638 13806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1: No bootstrap required, opened a new log
I20260924 04:05:38.819835 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.808100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33269 (local address 127.0.0.1:56352)
0924 04:05:38.809133 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.809146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.809240 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.809572 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.809575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.809728 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.809812 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.809818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.810869 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.810872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.811730 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.811739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.812447 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.812451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.812533 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.813039 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.813052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.814936 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.817085 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.817088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.817090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.817325 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.817700 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.817703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.817705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.817794 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.818381 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.818384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.818480 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.819311 (+ 831us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.819501 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970}
I20260924 04:05:38.821015 13881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:38.821302 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:05:38.821856 13881 log.cc:824] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:38.823583 13806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.823810 13806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:38.823859 13806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184cb5ac53bc4d6dac884e02424dd5e1, State: Initialized, Role: FOLLOWER
I20260924 04:05:38.824292 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846: No bootstrap required, opened a new log
I20260924 04:05:38.824393 13806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.825814 13806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260924 04:05:38.827589 13759 tablet.cc:2369] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:05:38 dist-test-slave-zcsk krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:05:38.828325 13881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.828583 13881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:38.828641 13881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bbc620951a845db823f8b1375c45846, State: Initialized, Role: FOLLOWER
I20260924 04:05:38.829176 13881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.830600 13881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:05:38.831883 13978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:05:38.832132 13978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:05:38.832597 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.821622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.0.98.61:33269)
0924 04:05:38.822315 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.822319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.822333 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:05:38.822366 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.822369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.822408 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.822479 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.823189 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.823880 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.825205 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.825502 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.825574 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.828787 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.828791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.828794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.828820 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.830349 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.830819 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.830823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.830825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.830920 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.831207 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.831210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.831211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.831426 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.831625 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.832328 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.832447 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619}
I20260924 04:05:38.836522 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.822162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.0.98.60:42055)
0924 04:05:38.822312 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.822316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.822328 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:05:38.822355 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.822358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.822403 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.822464 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.823618 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.824355 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.825394 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.825719 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.825782 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.832938 (+ 7156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.832942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.832945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.832974 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.834796 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.835363 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.835367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.835368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.835443 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.835670 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.835673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.835675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.835878 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.835975 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.836258 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.836373 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 04:05:38.836948 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.821885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42055 (local address 127.0.0.1:54026)
0924 04:05:38.822192 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:38.822206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:38.822290 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:38.823203 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:38.823206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:38.823398 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:38.823506 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.823512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.824485 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.824488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.825282 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:38.825292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.825919 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.825922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:38.826027 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.826447 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:38.826460 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:38.832763 (+ 6303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:38.834961 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.834964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.834966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.835268 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.835527 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:38.835530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:38.835531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:38.835598 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:38.836027 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:38.836030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:38.836112 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 04:05:38.836641 (+ 529us) client_negotiation.cc:240] Negotiation successful
0924 04:05:38.836782 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":133,"threads_started":1}
I20260924 04:05:38.837524 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:05:38.840065 13957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803: Bootstrap starting.
I20260924 04:05:38.838347 13813 master_runner.cc:186] Error getting master registration for 127.0.98.60:42055: OK, instance_id { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" instance_seqno: 1790222738782002 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:05:38.841876 13813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 04:05:38.842304 13991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:05:38.842375 13991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:05:38.842809 13957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:38.843714 13957 log.cc:824] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:38.832353 13738 master_runner.cc:186] Error getting master registration for 127.0.98.61:33269: OK, instance_id { permanent_uuid: "3bbc620951a845db823f8b1375c45846" instance_seqno: 1790222738551065 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:05:38.848121 13738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:05:38.828474 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.849104 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:05:38.849854 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.850085 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:05:38.852689 13957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803: No bootstrap required, opened a new log
I20260924 04:05:38.856601 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.840337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.0.98.60:42055)
0924 04:05:38.840428 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.840432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.844359 (+ 3927us) server_negotiation.cc:407] Connection header received
0924 04:05:38.844527 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.844530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.844577 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.844626 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.845553 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.846245 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.847415 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.847742 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.847819 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.850371 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.850376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.850378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.850405 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.852409 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.854032 (+ 1623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.854036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.854038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.854129 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.854463 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.854467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.854469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.854697 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.854830 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.856303 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.856437 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:05:38.857620 13957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.858006 13957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:38.858095 13957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8fd3289c7a49dda4367c8257535803, State: Initialized, Role: FOLLOWER
I20260924 04:05:38.858351 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.839695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.0.98.61:33269)
0924 04:05:38.839797 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.839802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.841298 (+ 1496us) server_negotiation.cc:407] Connection header received
0924 04:05:38.841459 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.841462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.841506 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.841552 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.842462 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.843180 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.844512 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.844844 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.844925 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.850774 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.850778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.850782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.850811 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.853166 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.855165 (+ 1999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.855169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.855171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.855262 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.855580 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.855583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.855585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.855803 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.855936 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.858048 (+ 2112us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.858184 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:05:38.858883 13957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.860529 13957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:05:38.866537 14012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:05:38.866851 14012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:05:38.867076 14001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.867304 14001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:05:38.867813 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:38.840980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.98.62:34363)
0924 04:05:38.841086 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:38.841090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:38.845110 (+ 4020us) server_negotiation.cc:407] Connection header received
0924 04:05:38.846104 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:38.846107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:38.846157 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:38.846202 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:38.860027 (+ 13825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.860566 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.861404 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:38.861756 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:38.861857 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:38.863739 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:38.863744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:38.863746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:38.863778 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:38.865496 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.866028 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.866032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.866034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.866126 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:38.866425 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:38.866428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:38.866430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:38.866660 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:38.866775 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:38.867520 (+ 745us) server_negotiation.cc:299] Negotiation successful
0924 04:05:38.867648 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:05:38.869257 13889 master_runner.cc:263] No leader master found from master 3d8fd3289c7a49dda4367c8257535803
I20260924 04:05:38.869329 13889 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:05:38.888633 13967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:38.888835 13967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.889918 13967 leader_election.cc:288] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bbc620951a845db823f8b1375c45846 (127.0.98.61:33269), 3d8fd3289c7a49dda4367c8257535803 (127.0.98.60:42055)
I20260924 04:05:38.890810 13855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bbc620951a845db823f8b1375c45846" is_pre_election: true
I20260924 04:05:38.891130 13855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184cb5ac53bc4d6dac884e02424dd5e1 in term 0.
I20260924 04:05:38.891579 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8fd3289c7a49dda4367c8257535803" is_pre_election: true
I20260924 04:05:38.891884 13931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184cb5ac53bc4d6dac884e02424dd5e1 in term 0.
I20260924 04:05:38.892422 13757 leader_election.cc:302] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184cb5ac53bc4d6dac884e02424dd5e1, 3bbc620951a845db823f8b1375c45846; no voters:
I20260924 04:05:38.892946 13967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:05:38.893036 13967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:05:38.893083 13967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:38.894802 13967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.895159 13967 leader_election.cc:288] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [CANDIDATE]: Term 1 election: Requested vote from peers 3bbc620951a845db823f8b1375c45846 (127.0.98.61:33269), 3d8fd3289c7a49dda4367c8257535803 (127.0.98.60:42055)
I20260924 04:05:38.895780 13855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bbc620951a845db823f8b1375c45846"
I20260924 04:05:38.896014 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8fd3289c7a49dda4367c8257535803"
I20260924 04:05:38.896136 13931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:38.897418 13931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184cb5ac53bc4d6dac884e02424dd5e1 in term 1.
I20260924 04:05:38.897742 13757 leader_election.cc:302] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184cb5ac53bc4d6dac884e02424dd5e1, 3d8fd3289c7a49dda4367c8257535803; no voters:
I20260924 04:05:38.897909 13855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:38.899169 13855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184cb5ac53bc4d6dac884e02424dd5e1 in term 1.
I20260924 04:05:38.900432 13967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:05:38.900701 13967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [term 1 LEADER]: Becoming Leader. State: Replica: 184cb5ac53bc4d6dac884e02424dd5e1, State: Running, Role: LEADER
I20260924 04:05:38.901075 13967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } }
I20260924 04:05:38.903256 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184cb5ac53bc4d6dac884e02424dd5e1. Latest consensus state: current_term: 1 leader_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.903393 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:38.903764 14016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:05:38.904829 14016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:05:38.915426 13931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [term 1 FOLLOWER]: Refusing update from remote peer 184cb5ac53bc4d6dac884e02424dd5e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:05:38.916236 14014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:05:38.921257 14001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184cb5ac53bc4d6dac884e02424dd5e1. Latest consensus state: current_term: 1 leader_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.921423 14001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:05:38.925038 13855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [term 1 FOLLOWER]: Refusing update from remote peer 184cb5ac53bc4d6dac884e02424dd5e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:05:38.925658 14014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:05:38.929687 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184cb5ac53bc4d6dac884e02424dd5e1. Latest consensus state: current_term: 1 leader_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.929824 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:05:38.935695 14014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.935858 14014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:38.938635 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.938786 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:05:38.939226 14017 mvcc.cc:205] Tried to move back new op lower bound from 7332752338589036544 to 7332752338543341568. Current Snapshot: MvccSnapshot[applied={T|T < 7332752338589036544}]
I20260924 04:05:38.942956 14020 mvcc.cc:205] Tried to move back new op lower bound from 7332752338589036544 to 7332752338543341568. Current Snapshot: MvccSnapshot[applied={T|T < 7332752338589036544}]
I20260924 04:05:38.944406 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.944552 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:38.945801 14001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184cb5ac53bc4d6dac884e02424dd5e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } peers { permanent_uuid: "3bbc620951a845db823f8b1375c45846" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33269 } } peers { permanent_uuid: "3d8fd3289c7a49dda4367c8257535803" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42055 } } }
I20260924 04:05:38.945930 14001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:05:38.950748 14018 mvcc.cc:205] Tried to move back new op lower bound from 7332752338589036544 to 7332752338543341568. Current Snapshot: MvccSnapshot[applied={T|T < 7332752338589036544}]
I20260924 04:05:38.953022 14016 catalog_manager.cc:1347] Generated new cluster ID: 44076e40672c4904bfb91ec689baa122
I20260924 04:05:38.953099 14016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:05:39.029947 14016 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:05:39.031239 14016 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:05:39.045390 14016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 184cb5ac53bc4d6dac884e02424dd5e1: Generated new TSK 0
I20260924 04:05:39.047215 14016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:05:39.843791 13991 catalog_manager.cc:1259] Loaded cluster ID: 44076e40672c4904bfb91ec689baa122
I20260924 04:05:39.843891 13991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846: loading cluster ID for follower catalog manager: success
I20260924 04:05:39.845685 13991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846: acquiring CA information for follower catalog manager: success
I20260924 04:05:39.846956 13991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bbc620951a845db823f8b1375c45846: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:05:39.868551 14012 catalog_manager.cc:1259] Loaded cluster ID: 44076e40672c4904bfb91ec689baa122
I20260924 04:05:39.868644 14012 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803: loading cluster ID for follower catalog manager: success
I20260924 04:05:39.870352 14012 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803: acquiring CA information for follower catalog manager: success
I20260924 04:05:39.871570 14012 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d8fd3289c7a49dda4367c8257535803: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:05:40.109531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 13738
I20260924 04:05:40.114933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 13813
I20260924 04:05:40.119758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 13889
2026-09-24T04:05:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3036 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:05:40 dist-test-slave-zcsk krb5kdc[14031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:05:40 dist-test-slave-zcsk krb5kdc[14031](info): set up 2 sockets
Sep 24 04:05:40 dist-test-slave-zcsk krb5kdc[14031](info): commencing operation
krb5kdc: starting...
W20260924 04:05:41.079950 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:05:41 dist-test-slave-zcsk krb5kdc[14031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:05:41.200156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:41.313031 14044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:41.313339 14044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:41.313400 14044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:41.317492 14044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:41.317561 14044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:41.317587 14044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:41.317610 14044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:41.317638 14044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:41.323169 14044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:41.324420 14044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:41.325486 14044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:41.336181 14049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:41.336553 14050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:41.336658 14044 server_base.cc:1034] running on GCE node
W20260924 04:05:41.336805 14052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:41.337519 14044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:41.339087 14044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:41.340283 14044 hybrid_clock.cc:648] HybridClock initialized: now 1790222741340256 us; error 38 us; skew 500 ppm
Sep 24 04:05:41 dist-test-slave-zcsk krb5kdc[14031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:41.344017 14044 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:05:41.345018 14044 webserver.cc:460] Webserver started at http://127.0.98.62:33549/ using document root <none> and password file <none>
I20260924 04:05:41.345837 14044 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:41.345979 14044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:41.346271 14044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:05:41.349262 14044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "f031ef95186844068f83a24d69f5fc4d"
format_stamp: "Formatted at 2026-09-24 04:05:41 on dist-test-slave-zcsk"
server_key: "ed2b4229378be780ff50a9dd476d61dd"
server_key_iv: "257c038f7f19ae486864d0131108ac93"
server_key_version: "encryptionkey@0"
I20260924 04:05:41.349969 14044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "f031ef95186844068f83a24d69f5fc4d"
format_stamp: "Formatted at 2026-09-24 04:05:41 on dist-test-slave-zcsk"
server_key: "ed2b4229378be780ff50a9dd476d61dd"
server_key_iv: "257c038f7f19ae486864d0131108ac93"
server_key_version: "encryptionkey@0"
I20260924 04:05:41.354794 14044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 04:05:41.358062 14059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:41.359782 14044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:05:41.360049 14044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "f031ef95186844068f83a24d69f5fc4d"
format_stamp: "Formatted at 2026-09-24 04:05:41 on dist-test-slave-zcsk"
server_key: "ed2b4229378be780ff50a9dd476d61dd"
server_key_iv: "257c038f7f19ae486864d0131108ac93"
server_key_version: "encryptionkey@0"
I20260924 04:05:41.360318 14044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:41.385423 14044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:41.388787 14044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:41.388965 14044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:41.395874 14044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39405
I20260924 04:05:41.395890 14111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39405 every 8 connection(s)
I20260924 04:05:41.396894 14044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:05:41.399638 14112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:41.405330 14112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d: Bootstrap starting.
I20260924 04:05:41.406683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 14044
I20260924 04:05:41.406805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:05:41.407069 392 external_mini_cluster.cc:1428] Setting key c70168031da1cdaad57a83f76d474bf7
I20260924 04:05:41.408207 14112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:41.409039 14112 log.cc:824] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:41.410831 14112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d: No bootstrap required, opened a new log
I20260924 04:05:41.414340 14112 raft_consensus.cc:348] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f031ef95186844068f83a24d69f5fc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } }
I20260924 04:05:41.414561 14112 raft_consensus.cc:374] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:41.414594 14112 raft_consensus.cc:729] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f031ef95186844068f83a24d69f5fc4d, State: Initialized, Role: FOLLOWER
I20260924 04:05:41.415087 14112 consensus_queue.cc:260] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f031ef95186844068f83a24d69f5fc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } }
I20260924 04:05:41.415266 14112 raft_consensus.cc:388] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:05:41.415335 14112 raft_consensus.cc:482] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 24 04:05:41 dist-test-slave-zcsk krb5kdc[14031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:41.415431 14112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:41.416404 14112 raft_consensus.cc:504] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f031ef95186844068f83a24d69f5fc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } }
I20260924 04:05:41.416764 14112 leader_election.cc:302] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f031ef95186844068f83a24d69f5fc4d; no voters:
I20260924 04:05:41.417109 14112 leader_election.cc:288] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:05:41.417207 14117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:05:41.417402 14117 raft_consensus.cc:686] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [term 1 LEADER]: Becoming Leader. State: Replica: f031ef95186844068f83a24d69f5fc4d, State: Running, Role: LEADER
I20260924 04:05:41.417708 14117 consensus_queue.cc:237] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f031ef95186844068f83a24d69f5fc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } }
I20260924 04:05:41.418460 14112 sys_catalog.cc:564] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:05:41.419394 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f031ef95186844068f83a24d69f5fc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f031ef95186844068f83a24d69f5fc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } } }
I20260924 04:05:41.419515 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:41.419862 14118 sys_catalog.cc:455] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f031ef95186844068f83a24d69f5fc4d. Latest consensus state: current_term: 1 leader_uuid: "f031ef95186844068f83a24d69f5fc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f031ef95186844068f83a24d69f5fc4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } } }
I20260924 04:05:41.419991 14118 sys_catalog.cc:458] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:41.419862 14126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:05:41.421525 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:41.408214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.0.98.62:39405)
0924 04:05:41.408664 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:41.408679 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:41.408715 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:05:41.409431 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:41.409450 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:41.409783 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:41.410090 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:41.410847 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:41.411832 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:41.412693 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:41.413156 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:41.413296 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:41.416017 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:41.416022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:41.416036 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:41.416104 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:41.417929 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:41.418320 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:41.418325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:41.418331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:41.418442 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:41.418729 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:41.418732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:41.418734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:41.419273 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:41.419496 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:41.420176 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 04:05:41.420512 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260924 04:05:41.423585 14126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:05:41.424467 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:05:41.429105 14126 catalog_manager.cc:1347] Generated new cluster ID: 8828e52a43564b8dac05641e107be0f3
I20260924 04:05:41.429171 14126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:05:41.439112 14126 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:05:41.440506 14126 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:05:41.447357 14126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f031ef95186844068f83a24d69f5fc4d: Generated new TSK 0
I20260924 04:05:41.448258 14126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:05:41.480353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 14044
2026-09-24T04:05:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1358 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:05:41 dist-test-slave-zcsk krb5kdc[14139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:05:41 dist-test-slave-zcsk krb5kdc[14139](info): set up 2 sockets
Sep 24 04:05:41 dist-test-slave-zcsk krb5kdc[14139](info): commencing operation
krb5kdc: starting...
W20260924 04:05:42.440078 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:05:42 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:05:42.559255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:42.667469 14152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:42.667765 14152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:42.667826 14152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:42.671872 14152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:42.671970 14152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:42.671998 14152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:42.672019 14152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:42.672039 14152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:42.677464 14152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:42.678556 14152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:42.679569 14152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:42.687784 14157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:42.687813 14158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:42.688028 14152 server_base.cc:1034] running on GCE node
W20260924 04:05:42.688089 14160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:42.688663 14152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:42.689780 14152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:42.690994 14152 hybrid_clock.cc:648] HybridClock initialized: now 1790222742690977 us; error 36 us; skew 500 ppm
Sep 24 04:05:42 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:42.697237 14152 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:05:42.698171 14152 webserver.cc:460] Webserver started at http://127.0.98.62:41489/ using document root <none> and password file <none>
I20260924 04:05:42.698716 14152 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:42.698784 14152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:42.698992 14152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:05:42.701076 14152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "22ad954a6a9b4919a67d8b8b7f4038ee"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "56c8d8fb412d8eb1bdbde8d8b4efe53c"
server_key_iv: "e3b214a2ce377cefe50aad74cfd2d0a7"
server_key_version: "encryptionkey@0"
I20260924 04:05:42.701609 14152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "22ad954a6a9b4919a67d8b8b7f4038ee"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "56c8d8fb412d8eb1bdbde8d8b4efe53c"
server_key_iv: "e3b214a2ce377cefe50aad74cfd2d0a7"
server_key_version: "encryptionkey@0"
I20260924 04:05:42.705312 14152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:05:42.707696 14167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:42.709004 14152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:05:42.709133 14152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "22ad954a6a9b4919a67d8b8b7f4038ee"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "56c8d8fb412d8eb1bdbde8d8b4efe53c"
server_key_iv: "e3b214a2ce377cefe50aad74cfd2d0a7"
server_key_version: "encryptionkey@0"
I20260924 04:05:42.709257 14152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:42.721515 14152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:42.724763 14152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:42.724951 14152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:42.731792 14152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37025
I20260924 04:05:42.732122 14219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37025 every 8 connection(s)
I20260924 04:05:42.732834 14152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:05:42.735188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 14152
I20260924 04:05:42.735291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:05:42.735555 392 external_mini_cluster.cc:1428] Setting key 7ce2f2d16b07a49b9797c2f29ec5cf16
I20260924 04:05:42.735790 14220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:42.743232 14220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Bootstrap starting.
Sep 24 04:05:42 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:42.745894 14220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:42.746675 14220 log.cc:824] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:42.748698 14220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: No bootstrap required, opened a new log
I20260924 04:05:42.750104 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:42.736940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.0.98.62:37025)
0924 04:05:42.737341 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:42.737358 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:42.737394 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:05:42.738055 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:42.738072 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:42.738421 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:42.738706 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:42.739781 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:42.740908 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:42.742161 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:42.742711 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:42.742867 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:42.745660 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:42.745665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:42.745680 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:42.745710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:42.747362 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:42.747779 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:42.747784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:42.747789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:42.747896 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:42.748187 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:42.748190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:42.748192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:42.748595 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:42.748787 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:42.749050 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 04:05:42.749338 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260924 04:05:42.752513 14220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:42.752761 14220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:42.752825 14220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ad954a6a9b4919a67d8b8b7f4038ee, State: Initialized, Role: FOLLOWER
I20260924 04:05:42.753253 14220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:42.753403 14220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:05:42.753474 14220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:05:42.753577 14220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:42.754559 14220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:42.754917 14220 leader_election.cc:302] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ad954a6a9b4919a67d8b8b7f4038ee; no voters:
I20260924 04:05:42.755239 14220 leader_election.cc:288] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:05:42.755386 14225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:05:42.755626 14225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 1 LEADER]: Becoming Leader. State: Replica: 22ad954a6a9b4919a67d8b8b7f4038ee, State: Running, Role: LEADER
I20260924 04:05:42.756043 14225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:42.756573 14220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:05:42.757751 14227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ad954a6a9b4919a67d8b8b7f4038ee. Latest consensus state: current_term: 1 leader_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:05:42.757902 14227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:42.758306 14226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:05:42.759135 14226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: This master's current role is: LEADER
W20260924 04:05:42.761518 14240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:05:42.761590 14240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:05:42.761649 14234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:05:42.762519 14234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:05:42.768290 14234 catalog_manager.cc:1347] Generated new cluster ID: 355c00b60d244d38b2dbd03117d53655
I20260924 04:05:42.768384 14234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:05:42.787468 14234 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:05:42.788470 14234 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:05:42.794675 14234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Generated new TSK 0
I20260924 04:05:42.795357 14234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:05:42.826273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:42.937834 14246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:42.938115 14246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:42.938164 14246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:42.942220 14246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:42.942291 14246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:42.942355 14246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:05:42.947773 14246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:42.948863 14246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:42.949841 14246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:42.959447 14252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:42.959465 14251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:42.959553 14246 server_base.cc:1034] running on GCE node
W20260924 04:05:42.959460 14254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:42.960158 14246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:42.960876 14246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:42.962069 14246 hybrid_clock.cc:648] HybridClock initialized: now 1790222742962038 us; error 50 us; skew 500 ppm
Sep 24 04:05:42 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:42.965129 14246 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:05:42.966135 14246 webserver.cc:460] Webserver started at http://127.0.98.1:44007/ using document root <none> and password file <none>
I20260924 04:05:42.966706 14246 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:42.966770 14246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:42.966967 14246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:05:42.969172 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "095e15e59a70454aa39f29f9724ecceb"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "ad4cb99ce7b3fa4f0837fba8cbc7eee1"
server_key_iv: "0e16aa62669ce3207d0a9579d3dc112c"
server_key_version: "encryptionkey@0"
I20260924 04:05:42.969705 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "095e15e59a70454aa39f29f9724ecceb"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "ad4cb99ce7b3fa4f0837fba8cbc7eee1"
server_key_iv: "0e16aa62669ce3207d0a9579d3dc112c"
server_key_version: "encryptionkey@0"
I20260924 04:05:42.973552 14246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:05:42.976146 14261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:42.977571 14246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:42.977684 14246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "095e15e59a70454aa39f29f9724ecceb"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "ad4cb99ce7b3fa4f0837fba8cbc7eee1"
server_key_iv: "0e16aa62669ce3207d0a9579d3dc112c"
server_key_version: "encryptionkey@0"
I20260924 04:05:42.977797 14246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:42.991747 14246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:42.994877 14246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:42.995059 14246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:42.995672 14246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:42.996553 14246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:42.996600 14246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:42.996644 14246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:42.996671 14246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:43.005445 14246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36577
I20260924 04:05:43.005470 14374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36577 every 8 connection(s)
I20260924 04:05:43.006371 14246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:05:43.012797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 14246
I20260924 04:05:43.012923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:05:43.013198 392 external_mini_cluster.cc:1428] Setting key 876693b6cd99d065221dd182e1edc4cb
Sep 24 04:05:43 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:43.020749 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.008412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57695 (local address 127.0.98.62:37025)
0924 04:05:43.008566 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.008569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.009492 (+ 923us) server_negotiation.cc:407] Connection header received
0924 04:05:43.010816 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.010820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.010870 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.010947 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:43.012277 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.012857 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.014297 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.014561 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.014660 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.017214 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:43.017217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:43.017219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:43.017245 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:43.018709 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:43.019238 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:43.019241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:43.019242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:43.019317 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:43.019626 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:43.019629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:43.019631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:43.019813 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:43.019899 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.020448 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.020582 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:05:43.021597 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.008806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:57695)
0924 04:05:43.009315 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.009362 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:43.010600 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:43.011145 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:43.011152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:43.011567 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:43.012135 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.012151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.013024 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.013028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.014172 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.014183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.014718 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.014721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.014881 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.015458 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:43.015475 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:43.017042 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:43.018821 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:43.018828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:43.018845 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:43.019140 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:43.019397 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:43.019400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:43.019402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:43.019541 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:43.019951 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:43.019958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:43.020209 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 04:05:43.020384 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 04:05:43.020735 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":131,"threads_started":1}
I20260924 04:05:43.022831 14375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:05:43.023079 14375 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:43.023603 14375 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:05:43.025202 14184 ts_manager.cc:194] Registered new tserver with Master: 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577)
I20260924 04:05:43.026775 14184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:05:43.039788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:43.151739 14380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:43.152092 14380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:43.152144 14380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:43.156445 14380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:43.156514 14380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:43.156582 14380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:05:43.162212 14380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:43.163362 14380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:43.164532 14380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:43.174528 14386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:43.174528 14388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:43.174533 14385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:43.174612 14380 server_base.cc:1034] running on GCE node
I20260924 04:05:43.175215 14380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:43.175884 14380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:43.177096 14380 hybrid_clock.cc:648] HybridClock initialized: now 1790222743177076 us; error 32 us; skew 500 ppm
Sep 24 04:05:43 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:43.180001 14380 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:05:43.180950 14380 webserver.cc:460] Webserver started at http://127.0.98.2:38349/ using document root <none> and password file <none>
I20260924 04:05:43.181551 14380 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:43.181613 14380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:43.181810 14380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:05:43.183972 14380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "59406b238da1406cb9d79aafc88a1912"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "4598b187adcb63e0b4b3d766fdf8528f"
server_key_iv: "57f830bce2a1c1e4913f318f9f6e2edb"
server_key_version: "encryptionkey@0"
I20260924 04:05:43.184522 14380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "59406b238da1406cb9d79aafc88a1912"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "4598b187adcb63e0b4b3d766fdf8528f"
server_key_iv: "57f830bce2a1c1e4913f318f9f6e2edb"
server_key_version: "encryptionkey@0"
I20260924 04:05:43.188472 14380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:05:43.191025 14395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:43.192488 14380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:43.192596 14380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "59406b238da1406cb9d79aafc88a1912"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "4598b187adcb63e0b4b3d766fdf8528f"
server_key_iv: "57f830bce2a1c1e4913f318f9f6e2edb"
server_key_version: "encryptionkey@0"
I20260924 04:05:43.192709 14380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:43.208904 14380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:43.212190 14380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:43.212376 14380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:43.212975 14380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:43.213809 14380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:43.213856 14380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:43.213909 14380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:43.213943 14380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:43.222347 14380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40387
I20260924 04:05:43.222379 14508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40387 every 8 connection(s)
I20260924 04:05:43.223285 14380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:05:43.226186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 14380
I20260924 04:05:43.226320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:05:43.226650 392 external_mini_cluster.cc:1428] Setting key 6fb29bad87e149ca9e99fd4cd7d278a5
Sep 24 04:05:43 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:05:43.239553 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.225260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40623 (local address 127.0.98.62:37025)
0924 04:05:43.225391 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.225394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.226405 (+ 1011us) server_negotiation.cc:407] Connection header received
0924 04:05:43.228037 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.228041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.228093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.228167 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:43.229862 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.230478 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.232259 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.232495 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.232586 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.235760 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:43.235764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:43.235766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:43.235796 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:43.237269 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:43.237858 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:43.237861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:43.237862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:43.238002 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:43.238331 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:43.238334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:43.238335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:43.238567 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:43.238641 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.239259 (+ 618us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.239387 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:05:43.240658 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.225668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:40623)
0924 04:05:43.226225 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.226289 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:43.227795 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:43.228361 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:43.228371 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:43.228932 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:43.229644 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.229665 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.230628 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.230632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.232125 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.232138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.232672 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.232676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.232903 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.233574 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:43.233597 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:43.235508 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:43.237426 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:43.237432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:43.237448 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:43.237764 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:43.238086 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:43.238088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:43.238090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:43.238236 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:43.238666 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:43.238672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:43.238963 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 04:05:43.239144 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 04:05:43.239498 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":122,"threads_started":1}
I20260924 04:05:43.242113 14509 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:05:43.242398 14509 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:43.242973 14509 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:05:43.244319 14184 ts_manager.cc:194] Registered new tserver with Master: 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
I20260924 04:05:43.244942 14184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:05:43.260073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:43.374089 14514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:43.374397 14514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:43.374444 14514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:43.378711 14514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:43.378793 14514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:43.378857 14514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:05:43.384543 14514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:43.385731 14514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:43.386792 14514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:43.396414 14519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:43.396637 14514 server_base.cc:1034] running on GCE node
W20260924 04:05:43.396414 14520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:43.396512 14522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:43.397177 14514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:43.397900 14514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:43.399073 14514 hybrid_clock.cc:648] HybridClock initialized: now 1790222743399053 us; error 32 us; skew 500 ppm
Sep 24 04:05:43 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:43.401889 14514 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:05:43.402793 14514 webserver.cc:460] Webserver started at http://127.0.98.3:36603/ using document root <none> and password file <none>
I20260924 04:05:43.403334 14514 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:43.403396 14514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:43.403599 14514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:05:43.405759 14514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "8eb72806452f45ab9ad9ed454df93f74"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "a43306c119d1967e8185ea177503b7d2"
server_key_iv: "1cc2d8105aeac4f478bfeed4deec3d09"
server_key_version: "encryptionkey@0"
I20260924 04:05:43.406349 14514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8eb72806452f45ab9ad9ed454df93f74"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "a43306c119d1967e8185ea177503b7d2"
server_key_iv: "1cc2d8105aeac4f478bfeed4deec3d09"
server_key_version: "encryptionkey@0"
I20260924 04:05:43.410367 14514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:05:43.412972 14529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:43.414290 14514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:05:43.414398 14514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "8eb72806452f45ab9ad9ed454df93f74"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "a43306c119d1967e8185ea177503b7d2"
server_key_iv: "1cc2d8105aeac4f478bfeed4deec3d09"
server_key_version: "encryptionkey@0"
I20260924 04:05:43.414515 14514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:43.435755 14514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:43.439275 14514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:43.439472 14514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:43.440091 14514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:43.440984 14514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:43.441041 14514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:43.441087 14514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:43.441102 14514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:43.449949 14514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38061
I20260924 04:05:43.450028 14642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38061 every 8 connection(s)
I20260924 04:05:43.450896 14514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:05:43.456822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 14514
I20260924 04:05:43.456931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:05:43.457155 392 external_mini_cluster.cc:1428] Setting key 8e192ceb33fbbc54abafc03d5f299df8
Sep 24 04:05:43 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:43.465240 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.452765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43569 (local address 127.0.98.62:37025)
0924 04:05:43.452920 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.452924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.453834 (+ 910us) server_negotiation.cc:407] Connection header received
0924 04:05:43.455169 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.455172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.455224 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.455300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:43.456646 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.457318 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.458725 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.458948 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.459037 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.461762 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:43.461766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:43.461768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:43.461796 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:43.463015 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:43.463548 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:43.463552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:43.463554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:43.463638 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:43.464073 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:43.464076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:43.464078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:43.464279 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:43.464378 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.464914 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.465056 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:05:43.466117 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.453159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:43569)
0924 04:05:43.453654 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.453700 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:43.454942 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:43.455469 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:43.455476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:43.455857 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:43.456489 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.456505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.457444 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.457448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.458613 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.458623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.459084 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.459088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.459272 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.459931 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:43.459955 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:43.461567 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:43.463136 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:43.463142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:43.463162 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:43.463444 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:43.463727 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:43.463730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:43.463732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:43.463963 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:43.464379 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:43.464387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:43.464663 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 04:05:43.464835 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 04:05:43.465214 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":146,"threads_started":1}
I20260924 04:05:43.467279 14643 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:05:43.467514 14643 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:43.468042 14643 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:05:43.469096 14184 ts_manager.cc:194] Registered new tserver with Master: 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061)
I20260924 04:05:43.469677 14184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43569
I20260924 04:05:43.471159 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:05:43.479871 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.473129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.0.98.62:37025)
0924 04:05:43.473245 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.473249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.473440 (+ 191us) server_negotiation.cc:407] Connection header received
0924 04:05:43.473530 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.473532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.473572 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.473637 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:43.474365 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.474860 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.475732 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.476030 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.476105 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.477333 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:43.477336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:43.477338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:43.477360 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:43.478644 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:43.478987 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:43.478989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:43.478990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:43.479054 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:43.479225 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:43.479226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:43.479227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:43.479421 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:43.479516 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.479668 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.479753 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:05:43.482959 14184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:05:43.486091 14184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:05:43.500727 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.494583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36577 (local address 127.0.0.1:39250)
0924 04:05:43.494991 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.495056 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:43.495193 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:43.496744 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:43.496747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:43.496767 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:43.496961 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.496969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.497936 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.497940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.499256 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.499267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.500136 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.500140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.500271 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.500322 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 04:05:43.500433 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:05:43.500571 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":136,"thread_start_us":76,"threads_started":1}
I20260924 04:05:43.500742 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.494750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.0.98.1:36577)
0924 04:05:43.496025 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.496039 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.496063 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:05:43.496143 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.496148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.496373 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.496570 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:43.497068 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.497813 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.499396 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.500025 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.500123 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.500293 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.500476 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.500578 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":118,"threads_started":1}
I20260924 04:05:43.501124 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.494613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38061 (local address 127.0.0.1:51390)
0924 04:05:43.495126 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.495138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:43.495202 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:43.496569 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:43.496575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:43.496598 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:43.496744 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.496751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.498276 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.498279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.499599 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.499609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.500722 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.500725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.500812 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.500834 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:05:43.500881 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:05:43.500980 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":48,"spinlock_wait_cycles":10624,"thread_start_us":101,"threads_started":1}
I20260924 04:05:43.501256 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.495331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40387 (local address 127.0.0.1:49018)
0924 04:05:43.495631 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.496079 (+ 448us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:43.496180 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:43.496739 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:43.496741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:43.496781 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:43.496923 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.496930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.498277 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.498280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.499601 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.499609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.500886 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.500890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.501004 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.501024 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:05:43.501093 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:05:43.501154 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":127,"thread_start_us":92,"threads_started":1}
I20260924 04:05:43.501396 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.495555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.0.98.2:40387)
0924 04:05:43.495879 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.495892 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.496153 (+ 261us) server_negotiation.cc:407] Connection header received
0924 04:05:43.496282 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.496286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.496443 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.496592 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:43.497039 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.498158 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.499733 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.500588 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.500691 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.500831 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.501148 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.501255 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":139,"threads_started":1}
I20260924 04:05:43.501809 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.495596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.98.3:38061)
0924 04:05:43.496131 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.496142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.496160 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:05:43.496217 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.496221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.496372 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.496570 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:43.496872 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.498087 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.499740 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.500589 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.501458 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.501568 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.501606 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.501674 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":146,"threads_started":1}
I20260924 04:05:43.504208 14309 tablet_service.cc:1467] Processing CreateTablet for tablet f0c53db6673d4e11a135f4038acbeb12 (DEFAULT_TABLE table=test-table [id=714e040e03934fef9714f3fb73f2206e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:43.504379 14577 tablet_service.cc:1467] Processing CreateTablet for tablet f0c53db6673d4e11a135f4038acbeb12 (DEFAULT_TABLE table=test-table [id=714e040e03934fef9714f3fb73f2206e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:43.504379 14443 tablet_service.cc:1467] Processing CreateTablet for tablet f0c53db6673d4e11a135f4038acbeb12 (DEFAULT_TABLE table=test-table [id=714e040e03934fef9714f3fb73f2206e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:43.505240 14309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c53db6673d4e11a135f4038acbeb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:43.505426 14577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c53db6673d4e11a135f4038acbeb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:43.505420 14443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c53db6673d4e11a135f4038acbeb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:43.512316 14660 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Bootstrap starting.
I20260924 04:05:43.514174 14662 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: Bootstrap starting.
I20260924 04:05:43.514693 14660 tablet_bootstrap.cc:654] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:43.515712 14660 log.cc:824] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:43.516039 14662 tablet_bootstrap.cc:654] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:43.516294 14661 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: Bootstrap starting.
I20260924 04:05:43.517462 14662 log.cc:824] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:43.517750 14660 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: No bootstrap required, opened a new log
I20260924 04:05:43.517959 14660 ts_tablet_manager.cc:1397] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:05:43.519021 14661 tablet_bootstrap.cc:654] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:43.519826 14662 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: No bootstrap required, opened a new log
I20260924 04:05:43.519976 14661 log.cc:824] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:43.520051 14662 ts_tablet_manager.cc:1397] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260924 04:05:43.521310 14660 raft_consensus.cc:348] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.521569 14660 raft_consensus.cc:374] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:43.521648 14660 raft_consensus.cc:729] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095e15e59a70454aa39f29f9724ecceb, State: Initialized, Role: FOLLOWER
I20260924 04:05:43.522142 14661 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: No bootstrap required, opened a new log
I20260924 04:05:43.522109 14660 consensus_queue.cc:260] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.522344 14661 ts_tablet_manager.cc:1397] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:05:43.522716 14375 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:05:43.523027 14660 ts_tablet_manager.cc:1428] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:05:43.523862 14662 raft_consensus.cc:348] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.524566 14662 raft_consensus.cc:374] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:43.524680 14662 raft_consensus.cc:729] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb72806452f45ab9ad9ed454df93f74, State: Initialized, Role: FOLLOWER
I20260924 04:05:43.525199 14662 consensus_queue.cc:260] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.525920 14643 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:05:43.526368 14662 ts_tablet_manager.cc:1428] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:05:43.527561 14661 raft_consensus.cc:348] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.527899 14661 raft_consensus.cc:374] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:43.527997 14661 raft_consensus.cc:729] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59406b238da1406cb9d79aafc88a1912, State: Initialized, Role: FOLLOWER
I20260924 04:05:43.528427 14661 consensus_queue.cc:260] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.529156 14509 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:05:43.529418 14661 ts_tablet_manager.cc:1428] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:05:43.582535 14667 raft_consensus.cc:482] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:43.582754 14667 raft_consensus.cc:504] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.583855 14667 leader_election.cc:288] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577), 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
I20260924 04:05:43.589144 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.584299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40667 (local address 127.0.98.1:36577)
0924 04:05:43.584422 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.584426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.584644 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:05:43.584728 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.584731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.584781 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.584851 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:43.585246 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.586048 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.587803 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.588457 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.588565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.588631 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.588930 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.589008 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:05:43.589144 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.584260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60151 (local address 127.0.98.2:40387)
0924 04:05:43.584402 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.584406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.584492 (+ 86us) server_negotiation.cc:407] Connection header received
0924 04:05:43.584626 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.584629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.584688 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.584787 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:43.585246 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.586064 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.587803 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.588457 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.588565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.588631 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.588949 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.589023 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:05:43.589170 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.584160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40387 (local address 127.0.98.3:60151)
0924 04:05:43.584403 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.584420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:43.584529 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:43.584880 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:43.584884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:43.584907 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:43.585092 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.585098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.586172 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.586176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.587663 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.587674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.588589 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.588593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.588779 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.588861 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 04:05:43.588924 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:05:43.589013 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":241,"spinlock_wait_cycles":12544}
I20260924 04:05:43.589184 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.584160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36577 (local address 127.0.98.3:40667)
0924 04:05:43.584563 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.584574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:43.584656 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:43.584925 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:43.584928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:43.584960 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:43.585076 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.585099 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.586172 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.586176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.587663 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:43.587674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.588589 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.588593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:43.588778 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.588822 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:05:43.588904 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:05:43.589011 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":157,"spinlock_wait_cycles":768,"thread_start_us":106,"threads_started":1}
I20260924 04:05:43.589903 14463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c53db6673d4e11a135f4038acbeb12" candidate_uuid: "8eb72806452f45ab9ad9ed454df93f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59406b238da1406cb9d79aafc88a1912" is_pre_election: true
I20260924 04:05:43.589991 14329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c53db6673d4e11a135f4038acbeb12" candidate_uuid: "8eb72806452f45ab9ad9ed454df93f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095e15e59a70454aa39f29f9724ecceb" is_pre_election: true
I20260924 04:05:43.590219 14463 raft_consensus.cc:2454] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb72806452f45ab9ad9ed454df93f74 in term 0.
I20260924 04:05:43.590255 14329 raft_consensus.cc:2454] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb72806452f45ab9ad9ed454df93f74 in term 0.
I20260924 04:05:43.590695 14532 leader_election.cc:302] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095e15e59a70454aa39f29f9724ecceb, 8eb72806452f45ab9ad9ed454df93f74; no voters:
I20260924 04:05:43.590971 14667 raft_consensus.cc:2781] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:05:43.591058 14667 raft_consensus.cc:482] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:05:43.591109 14667 raft_consensus.cc:3037] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:43.592340 14667 raft_consensus.cc:504] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.592732 14667 leader_election.cc:288] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [CANDIDATE]: Term 1 election: Requested vote from peers 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577), 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
I20260924 04:05:43.593094 14329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c53db6673d4e11a135f4038acbeb12" candidate_uuid: "8eb72806452f45ab9ad9ed454df93f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095e15e59a70454aa39f29f9724ecceb"
I20260924 04:05:43.593106 14463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c53db6673d4e11a135f4038acbeb12" candidate_uuid: "8eb72806452f45ab9ad9ed454df93f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59406b238da1406cb9d79aafc88a1912"
I20260924 04:05:43.593226 14329 raft_consensus.cc:3037] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:43.593226 14463 raft_consensus.cc:3037] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:43.594480 14329 raft_consensus.cc:2454] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb72806452f45ab9ad9ed454df93f74 in term 1.
I20260924 04:05:43.594480 14463 raft_consensus.cc:2454] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb72806452f45ab9ad9ed454df93f74 in term 1.
I20260924 04:05:43.594785 14532 leader_election.cc:302] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 095e15e59a70454aa39f29f9724ecceb, 59406b238da1406cb9d79aafc88a1912, 8eb72806452f45ab9ad9ed454df93f74; no voters:
I20260924 04:05:43.595005 14667 raft_consensus.cc:2781] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:05:43.595211 14667 raft_consensus.cc:686] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 LEADER]: Becoming Leader. State: Replica: 8eb72806452f45ab9ad9ed454df93f74, State: Running, Role: LEADER
I20260924 04:05:43.595530 14667 consensus_queue.cc:237] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:43.598963 14183 catalog_manager.cc:5495] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8eb72806452f45ab9ad9ed454df93f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } health_report { overall_health: UNKNOWN } } }
I20260924 04:05:43.625222 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.621752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.0.98.3:38061)
0924 04:05:43.621891 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:43.621895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:43.621909 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:05:43.621991 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:43.621994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:43.622055 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:43.622155 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:05:43.622393 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.622915 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.623750 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:43.623992 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:43.624070 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:43.624364 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:05:43.624848 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:05:43.624951 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:43.625054 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 04:05:43.625110 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:05:43.631433 14463 raft_consensus.cc:1261] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 1 FOLLOWER]: Refusing update from remote peer 8eb72806452f45ab9ad9ed454df93f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:05:43.631489 14329 raft_consensus.cc:1261] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Refusing update from remote peer 8eb72806452f45ab9ad9ed454df93f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:05:43.632390 14667 consensus_queue.cc:1035] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:05:43.632602 14670 consensus_queue.cc:1035] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:05:43.651573 14183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:05:43.651747 14183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:05:43.654510 14183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Sending DeleteTablet for 3 replicas of tablet f0c53db6673d4e11a135f4038acbeb12
I20260924 04:05:43.655381 14309 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c53db6673d4e11a135f4038acbeb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39250
I20260924 04:05:43.655453 14443 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c53db6673d4e11a135f4038acbeb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49018
I20260924 04:05:43.656064 14683 tablet_replica.cc:330] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: stopping tablet replica
I20260924 04:05:43.656042 14577 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c53db6673d4e11a135f4038acbeb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51390
I20260924 04:05:43.656454 14683 raft_consensus.cc:2229] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:05:43.656589 14685 tablet_replica.cc:330] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: stopping tablet replica
I20260924 04:05:43.656713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 14246
I20260924 04:05:43.656827 14683 raft_consensus.cc:2258] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:05:43.657002 14685 raft_consensus.cc:2229] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:05:43.657485 14685 raft_consensus.cc:2258] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:05:43.658715 14683 ts_tablet_manager.cc:1905] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:05:43.658732 14685 ts_tablet_manager.cc:1905] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:05:43.661470 14683 ts_tablet_manager.cc:1918] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:05:43.661492 14685 ts_tablet_manager.cc:1918] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:05:43.661566 14683 log.cc:1196] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/f0c53db6673d4e11a135f4038acbeb12
I20260924 04:05:43.661578 14685 log.cc:1196] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/f0c53db6673d4e11a135f4038acbeb12
I20260924 04:05:43.661885 14683 ts_tablet_manager.cc:1939] T f0c53db6673d4e11a135f4038acbeb12 P 59406b238da1406cb9d79aafc88a1912: Deleting consensus metadata
I20260924 04:05:43.661897 14685 ts_tablet_manager.cc:1939] T f0c53db6673d4e11a135f4038acbeb12 P 8eb72806452f45ab9ad9ed454df93f74: Deleting consensus metadata
I20260924 04:05:43.663293 14168 catalog_manager.cc:4841] TS 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387): tablet f0c53db6673d4e11a135f4038acbeb12 (table test-table [id=714e040e03934fef9714f3fb73f2206e]) successfully deleted
I20260924 04:05:43.663508 14168 catalog_manager.cc:4841] TS 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061): tablet f0c53db6673d4e11a135f4038acbeb12 (table test-table [id=714e040e03934fef9714f3fb73f2206e]) successfully deleted
W20260924 04:05:43.664585 14170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36577 (error 108)
I20260924 04:05:43.665184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 14380
I20260924 04:05:43.666046 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:43.665352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36577 (local address 127.0.0.1:39264)
0924 04:05:43.665887 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:43.665957 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451}
W20260924 04:05:43.670662 14170 catalog_manager.cc:4584] TS 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0c53db6673d4e11a135f4038acbeb12: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36577: connect: Connection refused (error 111)
I20260924 04:05:43.670733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 14514
I20260924 04:05:43.675602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 14152
I20260924 04:05:43.681061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--webserver_interface=127.0.98.62
--webserver_port=41489
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:43.793251 14687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:43.793542 14687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:43.793603 14687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:43.797636 14687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:43.797701 14687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:43.797734 14687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:43.797760 14687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:43.797789 14687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:43.803639 14687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:43.804965 14687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:43.806027 14687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:43.819370 14694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:43.819381 14696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:43.819380 14693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:43.819728 14687 server_base.cc:1034] running on GCE node
I20260924 04:05:43.820441 14687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:43.821717 14687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:43.822919 14687 hybrid_clock.cc:648] HybridClock initialized: now 1790222743822886 us; error 49 us; skew 500 ppm
Sep 24 04:05:43 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:43.826066 14687 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:05:43.827386 14687 webserver.cc:460] Webserver started at http://127.0.98.62:41489/ using document root <none> and password file <none>
I20260924 04:05:43.828107 14687 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:43.828193 14687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:43.832552 14687 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 04:05:43.834906 14703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:43.836329 14687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:43.836459 14687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "22ad954a6a9b4919a67d8b8b7f4038ee"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "56c8d8fb412d8eb1bdbde8d8b4efe53c"
server_key_iv: "e3b214a2ce377cefe50aad74cfd2d0a7"
server_key_version: "encryptionkey@0"
I20260924 04:05:43.837189 14687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:43.849733 14687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:43.852967 14687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:43.853157 14687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:43.860140 14687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37025
I20260924 04:05:43.860409 14755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37025 every 8 connection(s)
I20260924 04:05:43.861276 14687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:05:43.865929 14756 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:05:43.867681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 14687
I20260924 04:05:43.867808 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Bootstrap starting.
I20260924 04:05:43.868152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36577
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=44007
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:43.885375 14756 log.cc:824] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:43.894451 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:05:43.894852 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Bootstrap complete.
I20260924 04:05:43.898795 14756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:43.899221 14756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ad954a6a9b4919a67d8b8b7f4038ee, State: Initialized, Role: FOLLOWER
I20260924 04:05:43.899726 14756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:43.899860 14756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:05:43.899945 14756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:05:43.900048 14756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:05:43.901978 14756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:43.902318 14756 leader_election.cc:302] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ad954a6a9b4919a67d8b8b7f4038ee; no voters:
I20260924 04:05:43.902633 14756 leader_election.cc:288] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:05:43.902742 14760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:05:43.902947 14760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 2 LEADER]: Becoming Leader. State: Replica: 22ad954a6a9b4919a67d8b8b7f4038ee, State: Running, Role: LEADER
I20260924 04:05:43.903251 14760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:43.903944 14756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:05:43.904723 14761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:05:43.904760 14762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ad954a6a9b4919a67d8b8b7f4038ee. Latest consensus state: current_term: 2 leader_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:05:43.904848 14761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:43.904867 14762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:43.908984 14775 catalog_manager.cc:1259] Loaded cluster ID: 355c00b60d244d38b2dbd03117d53655
I20260924 04:05:43.909058 14775 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: loading cluster ID for follower catalog manager: success
W20260924 04:05:43.910041 14775 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260924 04:05:43.910120 14775 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260924 04:05:43.910176 14769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:05:43.912590 14769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:05:43.913086 14769 catalog_manager.cc:1259] Loaded cluster ID: 355c00b60d244d38b2dbd03117d53655
I20260924 04:05:43.913147 14769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:05:43.914191 14769 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 04:05:43.918934 14769 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 04:05:43.924449 14769 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:05:43.925249 14769 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Loaded TSK: 0
I20260924 04:05:43.926486 14769 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:05:43.983537 14758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:43.983846 14758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:43.983947 14758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:43.988185 14758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:43.988266 14758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:43.988354 14758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:05:43.994046 14758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44007
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:43.995126 14758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:43.996232 14758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:44.006434 14783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:44.006434 14784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:44.006544 14786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:44.006929 14758 server_base.cc:1034] running on GCE node
I20260924 04:05:44.007364 14758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:44.008081 14758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:44.009299 14758 hybrid_clock.cc:648] HybridClock initialized: now 1790222744009278 us; error 42 us; skew 500 ppm
Sep 24 04:05:44 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:44.012279 14758 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:05:44.013455 14758 webserver.cc:460] Webserver started at http://127.0.98.1:44007/ using document root <none> and password file <none>
I20260924 04:05:44.014181 14758 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:44.014285 14758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:44.021374 14758 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260924 04:05:44.024148 14793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:44.025632 14758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:05:44.025763 14758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "095e15e59a70454aa39f29f9724ecceb"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "ad4cb99ce7b3fa4f0837fba8cbc7eee1"
server_key_iv: "0e16aa62669ce3207d0a9579d3dc112c"
server_key_version: "encryptionkey@0"
I20260924 04:05:44.026376 14758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:44.052280 14758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:44.055387 14758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:44.055575 14758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:44.056206 14758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:44.057300 14800 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:05:44.061607 14758 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:05:44.061678 14758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 04:05:44.061756 14758 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:05:44.063510 14758 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:05:44.063561 14758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:05:44.063876 14800 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Bootstrap starting.
I20260924 04:05:44.072875 14758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36577
I20260924 04:05:44.073004 14907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36577 every 8 connection(s)
I20260924 04:05:44.073863 14758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:05:44.074957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 14758
I20260924 04:05:44.075335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40387
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=38349
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:44.083470 14800 log.cc:824] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:05:44 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:44.091934 14800 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:05:44.092298 14800 tablet_bootstrap.cc:492] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Bootstrap complete.
I20260924 04:05:44.093547 14800 ts_tablet_manager.cc:1397] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20260924 04:05:44.093798 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.076274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:49539)
0924 04:05:44.077097 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.077158 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:44.078263 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:44.079805 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:44.079812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:44.080251 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:44.080735 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.080749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.082673 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.082677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.084053 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.084063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.084869 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.084873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.085097 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.085809 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:44.085830 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:44.087755 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:44.090283 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:44.090290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:44.090304 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:44.090643 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:44.091043 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:44.091046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:44.091048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:44.091215 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:44.091892 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:44.091898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:44.092239 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 04:05:44.092462 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 04:05:44.092794 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":173,"threads_started":1}
I20260924 04:05:44.093834 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.076209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49539 (local address 127.0.98.62:37025)
0924 04:05:44.076722 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.076743 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.077288 (+ 545us) server_negotiation.cc:407] Connection header received
0924 04:05:44.078785 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.078818 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.079340 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.079747 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:44.080913 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.082489 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.084174 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.084710 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.084893 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.088031 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:44.088036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:44.088053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:44.088087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:44.090152 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.090775 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.090782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.090787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.090953 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.091317 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.091320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.091322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.091782 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:44.091952 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.092624 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.092934 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":124,"threads_started":1}
I20260924 04:05:44.096292 14800 raft_consensus.cc:348] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:44.096833 14800 raft_consensus.cc:729] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095e15e59a70454aa39f29f9724ecceb, State: Initialized, Role: FOLLOWER
I20260924 04:05:44.096925 14908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:05:44.097136 14908 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:44.097316 14800 consensus_queue.cc:260] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } } peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } }
I20260924 04:05:44.097679 14908 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:05:44.098326 14800 ts_tablet_manager.cc:1428] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:05:44.100152 14720 ts_manager.cc:194] Registered new tserver with Master: 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577)
I20260924 04:05:44.103427 14720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49539
I20260924 04:05:44.105893 14908 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
Sep 24 04:05:44 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:05:44.113835 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.102823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36577 (local address 127.0.0.1:39276)
0924 04:05:44.103602 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.103650 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:44.103794 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:44.104413 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:44.104418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:44.105505 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:44.105639 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.105647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.106852 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.106856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.107546 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.107556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.108289 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.108306 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.108390 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.108928 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:44.108939 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:44.110203 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:44.112210 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:44.112218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:44.112222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:44.112463 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:44.112770 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:44.112772 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:44.112773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:44.112841 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:44.113315 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:44.113319 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:44.113438 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 04:05:44.113545 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:05:44.113700 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":113,"threads_started":1}
I20260924 04:05:44.113983 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.102912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.0.98.1:36577)
0924 04:05:44.103305 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.103316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.103749 (+ 433us) server_negotiation.cc:407] Connection header received
0924 04:05:44.103970 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.103976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.104179 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.104315 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:44.105779 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.106751 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.107668 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.108187 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.108300 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.110339 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:44.110349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:44.110362 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:44.110388 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:44.112109 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.112574 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.112581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.112583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.112684 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.112910 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.112913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.112914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.113225 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:44.113340 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.113641 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.113814 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":294,"thread_start_us":91,"threads_started":1}
I20260924 04:05:44.115002 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c53db6673d4e11a135f4038acbeb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39276
I20260924 04:05:44.115813 14917 tablet_replica.cc:330] stopping tablet replica
I20260924 04:05:44.116125 14917 raft_consensus.cc:2229] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:05:44.116362 14917 raft_consensus.cc:2258] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:05:44.117285 14917 ts_tablet_manager.cc:1905] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:05:44.121138 14917 ts_tablet_manager.cc:1918] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:05:44.121232 14917 log.cc:1196] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/f0c53db6673d4e11a135f4038acbeb12
I20260924 04:05:44.121469 14917 ts_tablet_manager.cc:1939] T f0c53db6673d4e11a135f4038acbeb12 P 095e15e59a70454aa39f29f9724ecceb: Deleting consensus metadata
I20260924 04:05:44.123229 14706 catalog_manager.cc:4841] TS 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577): tablet f0c53db6673d4e11a135f4038acbeb12 (table test-table [id=714e040e03934fef9714f3fb73f2206e]) successfully deleted
W20260924 04:05:44.196364 14911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:44.196650 14911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:44.196722 14911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:44.200762 14911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:44.200829 14911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:44.200910 14911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:05:44.206465 14911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38349
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:44.207536 14911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:44.208621 14911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:44.218297 14923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:44.218391 14911 server_base.cc:1034] running on GCE node
W20260924 04:05:44.218297 14925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:44.218375 14922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:44.218961 14911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:44.219632 14911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:44.220804 14911 hybrid_clock.cc:648] HybridClock initialized: now 1790222744220781 us; error 36 us; skew 500 ppm
Sep 24 04:05:44 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:44.223493 14911 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:05:44.224396 14911 webserver.cc:460] Webserver started at http://127.0.98.2:38349/ using document root <none> and password file <none>
I20260924 04:05:44.224924 14911 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:44.224992 14911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:44.229050 14911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:05:44.231299 14932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:44.232736 14911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:05:44.232903 14911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "59406b238da1406cb9d79aafc88a1912"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "4598b187adcb63e0b4b3d766fdf8528f"
server_key_iv: "57f830bce2a1c1e4913f318f9f6e2edb"
server_key_version: "encryptionkey@0"
I20260924 04:05:44.233518 14911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:44.264891 14911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:44.268543 14911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:44.268715 14911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:44.269311 14911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:44.270180 14911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:44.270231 14911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:44.270273 14911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:44.270287 14911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:44.279048 14911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40387
I20260924 04:05:44.279121 15045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40387 every 8 connection(s)
I20260924 04:05:44.280205 14911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:05:44.284547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 14911
I20260924 04:05:44.284991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38061
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=36603
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:05:44 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:44.295611 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.282151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37131 (local address 127.0.98.62:37025)
0924 04:05:44.282281 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.282284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.283218 (+ 934us) server_negotiation.cc:407] Connection header received
0924 04:05:44.284593 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.284597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.284659 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.284742 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:44.286487 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.287363 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.288622 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.288991 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.289109 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.291807 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:44.291811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:44.291825 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:44.291862 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:44.293406 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.293937 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.293940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.293943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.294039 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.294337 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.294340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.294342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.294588 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:44.294691 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.295274 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.295435 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:05:44.296515 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.282542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:37131)
0924 04:05:44.283049 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.283099 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:44.284342 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:44.285046 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:44.285057 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:44.285566 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:44.286278 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.286302 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.287509 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.287512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.288492 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.288502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.289150 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.289153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.289330 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.289976 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:44.289993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:44.291611 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:44.293522 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:44.293526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:44.293542 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:44.293839 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:44.294122 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:44.294124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:44.294125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:44.294256 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:44.294716 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:44.294722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:44.295024 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 04:05:44.295205 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 04:05:44.295504 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":152,"threads_started":1}
I20260924 04:05:44.297750 15046 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:05:44.298039 15046 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:44.298623 15046 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:05:44.299798 14720 ts_manager.cc:194] Registered new tserver with Master: 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
I20260924 04:05:44.300581 14720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37131
W20260924 04:05:44.396647 15049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:44.396921 15049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:44.396963 15049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:44.400902 15049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:44.400959 15049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:44.401016 15049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:05:44.406447 15049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36603
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:44.407430 15049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:44.408433 15049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:05:44.417840 15049 server_base.cc:1034] running on GCE node
W20260924 04:05:44.417768 15054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:44.417809 15055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:44.417788 15057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:44.418534 15049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:44.419203 15049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:44.420373 15049 hybrid_clock.cc:648] HybridClock initialized: now 1790222744420350 us; error 39 us; skew 500 ppm
Sep 24 04:05:44 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:44.423161 15049 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:05:44.424163 15049 webserver.cc:460] Webserver started at http://127.0.98.3:36603/ using document root <none> and password file <none>
I20260924 04:05:44.424729 15049 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:44.424813 15049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:44.428977 15049 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:05:44.431207 15064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:44.432531 15049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:05:44.432648 15049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "8eb72806452f45ab9ad9ed454df93f74"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "a43306c119d1967e8185ea177503b7d2"
server_key_iv: "1cc2d8105aeac4f478bfeed4deec3d09"
server_key_version: "encryptionkey@0"
I20260924 04:05:44.433195 15049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:44.469426 15049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:44.472872 15049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:44.473057 15049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:44.473667 15049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:44.474536 15049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:44.474583 15049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:44.474637 15049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:44.474666 15049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:44.482993 15049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38061
I20260924 04:05:44.483073 15177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38061 every 8 connection(s)
I20260924 04:05:44.484017 15049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:05:44.492138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 15049
Sep 24 04:05:44 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:05:44.497936 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.485874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54115 (local address 127.0.98.62:37025)
0924 04:05:44.486026 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.486030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.486795 (+ 765us) server_negotiation.cc:407] Connection header received
0924 04:05:44.488146 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.488149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.488217 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.488284 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:44.489484 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.490018 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.491171 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.491412 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.491503 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.494234 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:44.494245 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:44.494248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:44.494276 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:44.495785 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.496393 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.496397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.496399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.496497 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.496864 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.496868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.496870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.497041 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:44.497124 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.497640 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.497760 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:05:44.498984 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.486156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:54115)
0924 04:05:44.486632 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.486676 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:44.487884 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:44.488480 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:44.488488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:44.488858 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:44.489338 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.489351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.490140 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.490144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.491050 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.491059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.491556 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.491560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.491735 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.492399 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:44.492421 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:44.494051 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:44.495930 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:44.495938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:44.495955 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:44.496295 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:44.496586 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:44.496588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:44.496589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:44.496760 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:44.497134 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:44.497141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:44.497396 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 04:05:44.497564 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:05:44.497915 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260924 04:05:44.500201 15178 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:05:44.500447 15178 heartbeater.cc:464] Registering TS with master...
I20260924 04:05:44.501024 15178 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:05:44.501106 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.492919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.0.98.62:37025)
0924 04:05:44.493179 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.493183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.493220 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:05:44.493348 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.493351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.493402 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.493503 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:44.494228 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.494761 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.495960 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.496316 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.496412 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.498137 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:44.498140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:44.498143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:44.498165 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:44.499590 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.500035 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.500037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.500039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.500111 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:44.500377 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:44.500380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:44.500382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:44.500561 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:44.500711 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.500899 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.500991 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260924 04:05:44.502202 14720 ts_manager.cc:194] Registered new tserver with Master: 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061)
I20260924 04:05:44.502863 14720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54115
I20260924 04:05:44.502947 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:05:44.507136 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.503505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.0.98.62:37025)
0924 04:05:44.503603 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.503607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.503825 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:05:44.503981 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.503984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.504075 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.504154 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:05:44.504377 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.504959 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.505807 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.506078 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.506167 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.506456 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:05:44.506803 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:05:44.506901 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.506967 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.507020 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:05:44.507759 14720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:05:44.511013 14720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:05:44.518395 14832 tablet_service.cc:1467] Processing CreateTablet for tablet 36bb5a8b1b37420e9621e5e070d3a388 (DEFAULT_TABLE table=test-table [id=b27b2403070245fdaa362d34b59e36f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:44.519071 14832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bb5a8b1b37420e9621e5e070d3a388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:44.522024 14800 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Bootstrap starting.
I20260924 04:05:44.522744 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.517397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38061 (local address 127.0.0.1:51404)
0924 04:05:44.517501 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.517516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:44.517586 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:44.518519 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:44.518522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:44.518540 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:44.518648 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.518653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.519958 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.519962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.521142 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.521152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.522293 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.522297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.522406 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.522451 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 04:05:44.522512 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:05:44.522594 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":193}
I20260924 04:05:44.522768 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.517585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.0.98.3:38061)
0924 04:05:44.518018 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.518030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.518054 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:05:44.518129 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.518135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.518371 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.518546 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:44.518740 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.519788 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.521268 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.522141 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.522346 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.522505 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.522556 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.522642 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":111,"threads_started":1}
I20260924 04:05:44.522928 14800 tablet_bootstrap.cc:654] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:44.523229 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.517607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40387 (local address 127.0.0.1:49024)
0924 04:05:44.517823 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.517837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:44.517916 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:44.518889 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:44.518891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:44.518908 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:44.519011 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.519017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.520005 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.520008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.521300 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.521312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.522293 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.522297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.523033 (+ 736us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.523053 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:05:44.523088 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 04:05:44.523135 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":24,"thread_start_us":92,"threads_started":1}
I20260924 04:05:44.524230 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.518035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.0.98.2:40387)
0924 04:05:44.518447 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.518461 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.518484 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:05:44.518559 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.518564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.518770 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.518902 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:44.519098 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.519821 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.521438 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.522175 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.523645 (+ 1470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.523815 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.523866 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.524100 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":106,"threads_started":1}
I20260924 04:05:44.524389 14800 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: No bootstrap required, opened a new log
I20260924 04:05:44.524468 14800 ts_tablet_manager.cc:1397] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:05:44.524901 14800 raft_consensus.cc:348] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.525013 14800 raft_consensus.cc:374] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:44.525046 14800 raft_consensus.cc:729] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095e15e59a70454aa39f29f9724ecceb, State: Initialized, Role: FOLLOWER
I20260924 04:05:44.525172 14800 consensus_queue.cc:260] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.525435 14800 ts_tablet_manager.cc:1428] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:05:44.526331 15112 tablet_service.cc:1467] Processing CreateTablet for tablet 36bb5a8b1b37420e9621e5e070d3a388 (DEFAULT_TABLE table=test-table [id=b27b2403070245fdaa362d34b59e36f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:44.526525 14980 tablet_service.cc:1467] Processing CreateTablet for tablet 36bb5a8b1b37420e9621e5e070d3a388 (DEFAULT_TABLE table=test-table [id=b27b2403070245fdaa362d34b59e36f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:05:44.527344 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bb5a8b1b37420e9621e5e070d3a388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:44.527493 14980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bb5a8b1b37420e9621e5e070d3a388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:05:44.533932 15191 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: Bootstrap starting.
I20260924 04:05:44.533953 15192 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: Bootstrap starting.
I20260924 04:05:44.536267 15192 tablet_bootstrap.cc:654] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:44.536650 15191 tablet_bootstrap.cc:654] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: Neither blocks nor log segments found. Creating new log.
I20260924 04:05:44.537379 15191 log.cc:824] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:44.537379 15192 log.cc:824] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:44.539265 15191 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: No bootstrap required, opened a new log
I20260924 04:05:44.539273 15192 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: No bootstrap required, opened a new log
I20260924 04:05:44.539459 15191 ts_tablet_manager.cc:1397] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:05:44.539460 15192 ts_tablet_manager.cc:1397] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:05:44.542388 15192 raft_consensus.cc:348] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.542380 15191 raft_consensus.cc:348] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.542609 15191 raft_consensus.cc:374] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:44.542613 15192 raft_consensus.cc:374] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:05:44.542641 15191 raft_consensus.cc:729] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59406b238da1406cb9d79aafc88a1912, State: Initialized, Role: FOLLOWER
I20260924 04:05:44.542678 15192 raft_consensus.cc:729] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb72806452f45ab9ad9ed454df93f74, State: Initialized, Role: FOLLOWER
I20260924 04:05:44.543067 15191 consensus_queue.cc:260] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.543075 15192 consensus_queue.cc:260] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.543926 15178 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:05:44.544067 15191 ts_tablet_manager.cc:1428] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:05:44.544070 15192 ts_tablet_manager.cc:1428] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:05:44.544401 15046 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
W20260924 04:05:44.575059 14909 tablet.cc:2369] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:05:44.577349 15196 raft_consensus.cc:482] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:44.577534 15196 raft_consensus.cc:504] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.578598 15196 leader_election.cc:288] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061), 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577)
I20260924 04:05:44.582019 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.578800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38061 (local address 127.0.98.2:59507)
0924 04:05:44.578947 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.578962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:44.579047 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:44.579320 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:44.579323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:44.579339 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:44.579447 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.579451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.580177 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.580180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.580992 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.581001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.581624 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.581627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.581727 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.581744 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:05:44.581815 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:05:44.581883 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":25}
I20260924 04:05:44.582019 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.579007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59507 (local address 127.0.98.3:38061)
0924 04:05:44.579123 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.579127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.579140 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:05:44.579182 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.579184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.579224 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.579293 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:44.579553 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.580082 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.581113 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.581514 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.581603 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.581665 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.581838 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.581904 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:05:44.582360 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.578925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48773 (local address 127.0.98.1:36577)
0924 04:05:44.579042 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.579046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.579271 (+ 225us) server_negotiation.cc:407] Connection header received
0924 04:05:44.579385 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.579387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.579440 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.579518 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:05:44.579761 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.580342 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.581374 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.581798 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.581887 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.582049 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.582168 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.582245 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:05:44.582670 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.578820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36577 (local address 127.0.98.2:48773)
0924 04:05:44.579179 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.579189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:44.579284 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:44.579542 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:44.579545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:44.579559 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:05:44.579661 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.579666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.580447 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.580450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.581258 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:44.581266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.581910 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.581913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:44.582014 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.582034 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:05:44.582493 (+ 459us) client_negotiation.cc:240] Negotiation successful
0924 04:05:44.582567 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":27,"thread_start_us":135,"threads_started":1}
I20260924 04:05:44.582670 15132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bb5a8b1b37420e9621e5e070d3a388" candidate_uuid: "59406b238da1406cb9d79aafc88a1912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb72806452f45ab9ad9ed454df93f74" is_pre_election: true
I20260924 04:05:44.582998 15132 raft_consensus.cc:2454] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59406b238da1406cb9d79aafc88a1912 in term 0.
I20260924 04:05:44.583236 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bb5a8b1b37420e9621e5e070d3a388" candidate_uuid: "59406b238da1406cb9d79aafc88a1912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095e15e59a70454aa39f29f9724ecceb" is_pre_election: true
I20260924 04:05:44.583487 14933 leader_election.cc:302] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59406b238da1406cb9d79aafc88a1912, 8eb72806452f45ab9ad9ed454df93f74; no voters:
I20260924 04:05:44.583526 14862 raft_consensus.cc:2454] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59406b238da1406cb9d79aafc88a1912 in term 0.
I20260924 04:05:44.583817 15196 raft_consensus.cc:2781] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:05:44.583933 15196 raft_consensus.cc:482] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:05:44.583989 15196 raft_consensus.cc:3037] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:44.584969 15196 raft_consensus.cc:504] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.585286 15196 leader_election.cc:288] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061), 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577)
I20260924 04:05:44.585602 15132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bb5a8b1b37420e9621e5e070d3a388" candidate_uuid: "59406b238da1406cb9d79aafc88a1912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb72806452f45ab9ad9ed454df93f74"
I20260924 04:05:44.585602 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bb5a8b1b37420e9621e5e070d3a388" candidate_uuid: "59406b238da1406cb9d79aafc88a1912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095e15e59a70454aa39f29f9724ecceb"
I20260924 04:05:44.585721 14862 raft_consensus.cc:3037] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:44.585722 15132 raft_consensus.cc:3037] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:05:44.586962 14862 raft_consensus.cc:2454] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59406b238da1406cb9d79aafc88a1912 in term 1.
I20260924 04:05:44.586962 15132 raft_consensus.cc:2454] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59406b238da1406cb9d79aafc88a1912 in term 1.
I20260924 04:05:44.587248 14935 leader_election.cc:302] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095e15e59a70454aa39f29f9724ecceb, 59406b238da1406cb9d79aafc88a1912; no voters:
I20260924 04:05:44.587451 15196 raft_consensus.cc:2781] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:05:44.587649 15196 raft_consensus.cc:686] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 1 LEADER]: Becoming Leader. State: Replica: 59406b238da1406cb9d79aafc88a1912, State: Running, Role: LEADER
I20260924 04:05:44.587991 15196 consensus_queue.cc:237] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:44.591091 14720 catalog_manager.cc:5495] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59406b238da1406cb9d79aafc88a1912 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "59406b238da1406cb9d79aafc88a1912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } health_report { overall_health: UNKNOWN } } }
I20260924 04:05:44.613454 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.609897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.0.98.2:40387)
0924 04:05:44.610030 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:44.610035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:44.610049 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:05:44.610182 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:44.610184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:44.610245 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:44.610338 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:05:44.610581 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.611123 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.611935 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:44.612150 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:44.612236 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:44.612531 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:05:44.612962 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:05:44.613070 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:44.613225 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 04:05:44.613291 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:05:44.619894 15132 raft_consensus.cc:1261] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 FOLLOWER]: Refusing update from remote peer 59406b238da1406cb9d79aafc88a1912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:05:44.619892 14862 raft_consensus.cc:1261] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Refusing update from remote peer 59406b238da1406cb9d79aafc88a1912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:05:44.620733 15196 consensus_queue.cc:1035] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:05:44.620972 15198 consensus_queue.cc:1035] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:05:44.626154 15201 mvcc.cc:205] Tried to move back new op lower bound from 7332752361955799040 to 7332752361835372544. Current Snapshot: MvccSnapshot[applied={T|T < 7332752361955799040}]
I20260924 04:05:44.627491 15202 mvcc.cc:205] Tried to move back new op lower bound from 7332752361955799040 to 7332752361835372544. Current Snapshot: MvccSnapshot[applied={T|T < 7332752361955799040}]
I20260924 04:05:44.635932 14719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:05:44.636114 14719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:05:44.638664 14719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Sending DeleteTablet for 3 replicas of tablet 36bb5a8b1b37420e9621e5e070d3a388
I20260924 04:05:44.639438 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet 36bb5a8b1b37420e9621e5e070d3a388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39276
I20260924 04:05:44.639439 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet 36bb5a8b1b37420e9621e5e070d3a388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51404
I20260924 04:05:44.639482 14980 tablet_service.cc:1514] Processing DeleteTablet for tablet 36bb5a8b1b37420e9621e5e070d3a388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:05:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49024
I20260924 04:05:44.639886 15210 tablet_replica.cc:330] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: stopping tablet replica
I20260924 04:05:44.640008 15212 tablet_replica.cc:330] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: stopping tablet replica
I20260924 04:05:44.640105 15210 raft_consensus.cc:2229] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:05:44.640265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 14758
I20260924 04:05:44.640288 15210 raft_consensus.cc:2258] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:05:44.641394 15211 tablet_replica.cc:330] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: stopping tablet replica
I20260924 04:05:44.641713 15212 raft_consensus.cc:2229] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:05:44.641719 15211 raft_consensus.cc:2229] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:05:44.642032 15212 raft_consensus.cc:2258] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:05:44.642083 15211 raft_consensus.cc:2258] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:05:44.643420 15212 ts_tablet_manager.cc:1905] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:05:44.643579 15211 ts_tablet_manager.cc:1905] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:05:44.646175 15212 ts_tablet_manager.cc:1918] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:05:44.646370 15212 log.cc:1196] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/36bb5a8b1b37420e9621e5e070d3a388
I20260924 04:05:44.646437 15211 ts_tablet_manager.cc:1918] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:05:44.646660 15211 log.cc:1196] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/36bb5a8b1b37420e9621e5e070d3a388
I20260924 04:05:44.646788 15212 ts_tablet_manager.cc:1939] T 36bb5a8b1b37420e9621e5e070d3a388 P 8eb72806452f45ab9ad9ed454df93f74: Deleting consensus metadata
I20260924 04:05:44.647078 15211 ts_tablet_manager.cc:1939] T 36bb5a8b1b37420e9621e5e070d3a388 P 59406b238da1406cb9d79aafc88a1912: Deleting consensus metadata
W20260924 04:05:44.647661 14706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36577 (error 108)
I20260924 04:05:44.648202 14704 catalog_manager.cc:4841] TS 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061): tablet 36bb5a8b1b37420e9621e5e070d3a388 (table test-table [id=b27b2403070245fdaa362d34b59e36f0]) successfully deleted
I20260924 04:05:44.648396 14704 catalog_manager.cc:4841] TS 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387): tablet 36bb5a8b1b37420e9621e5e070d3a388 (table test-table [id=b27b2403070245fdaa362d34b59e36f0]) successfully deleted
I20260924 04:05:44.648887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 14911
I20260924 04:05:44.651312 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:44.651032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36577 (local address 127.0.0.1:39280)
0924 04:05:44.651167 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:44.651237 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 04:05:44.651466 14706 catalog_manager.cc:4584] TS 095e15e59a70454aa39f29f9724ecceb (127.0.98.1:36577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36bb5a8b1b37420e9621e5e070d3a388: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36577: connect: Connection refused (error 111)
I20260924 04:05:44.654507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 15049
I20260924 04:05:44.659250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 14687
I20260924 04:05:44.664582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--webserver_interface=127.0.98.62
--webserver_port=41489
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:05:44.774350 15214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:44.774653 15214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:44.774714 15214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:44.778844 15214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:05:44.778925 15214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:44.778954 15214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:44.778975 15214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:05:44.778996 15214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:05:44.784583 15214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:44.785781 15214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:44.786834 15214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:44.795502 15220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:44.795487 15219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:44.795553 15214 server_base.cc:1034] running on GCE node
W20260924 04:05:44.795487 15222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:44.796397 15214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:44.797505 15214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:44.798671 15214 hybrid_clock.cc:648] HybridClock initialized: now 1790222744798641 us; error 43 us; skew 500 ppm
Sep 24 04:05:44 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:44.801542 15214 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:05:44.802419 15214 webserver.cc:460] Webserver started at http://127.0.98.62:41489/ using document root <none> and password file <none>
I20260924 04:05:44.802956 15214 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:44.803022 15214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:44.807615 15214 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:05:44.809942 15229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:44.811327 15214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:05:44.811450 15214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "22ad954a6a9b4919a67d8b8b7f4038ee"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "56c8d8fb412d8eb1bdbde8d8b4efe53c"
server_key_iv: "e3b214a2ce377cefe50aad74cfd2d0a7"
server_key_version: "encryptionkey@0"
I20260924 04:05:44.812126 15214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:44.829569 15214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:44.833370 15214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:44.833585 15214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:44.840482 15214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37025
I20260924 04:05:44.840811 15281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37025 every 8 connection(s)
I20260924 04:05:44.841601 15214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:05:44.846196 15282 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:05:44.848019 15282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Bootstrap starting.
I20260924 04:05:44.850914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 15214
I20260924 04:05:44.851361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36577
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=44007
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:44.865774 15282 log.cc:824] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:44.877596 15282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:05:44.877967 15282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee: Bootstrap complete.
I20260924 04:05:44.881807 15282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:44.882192 15282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ad954a6a9b4919a67d8b8b7f4038ee, State: Initialized, Role: FOLLOWER
I20260924 04:05:44.882660 15282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:44.882777 15282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:05:44.882841 15282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:05:44.882933 15282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:05:44.884822 15282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:44.885128 15282 leader_election.cc:302] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ad954a6a9b4919a67d8b8b7f4038ee; no voters:
I20260924 04:05:44.885489 15282 leader_election.cc:288] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:05:44.885562 15286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:05:44.885749 15286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [term 3 LEADER]: Becoming Leader. State: Replica: 22ad954a6a9b4919a67d8b8b7f4038ee, State: Running, Role: LEADER
I20260924 04:05:44.886040 15286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:05:44.886727 15282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:05:44.886956 15288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ad954a6a9b4919a67d8b8b7f4038ee. Latest consensus state: current_term: 3 leader_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:05:44.887079 15288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:44.887676 15287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ad954a6a9b4919a67d8b8b7f4038ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:05:44.887758 15287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ad954a6a9b4919a67d8b8b7f4038ee [sys.catalog]: This master's current role is: LEADER
I20260924 04:05:44.888571 15290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:05:44.893767 15290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:05:44.894399 15290 catalog_manager.cc:1259] Loaded cluster ID: 355c00b60d244d38b2dbd03117d53655
I20260924 04:05:44.894460 15290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 04:05:44.895498 15290 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe840c7e9ad google::LogMessage::Fail() at ??:0
@ 0x7fe840c8188a google::LogMessage::SendToLog() at ??:0
@ 0x7fe840c7e4de google::LogMessage::Flush() at ??:0
@ 0x7fe840c7fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe84b5c1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe84b5c2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe84b5bf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe84b603640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe842ed03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe842ed0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe842ed23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe842ec1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe8445056db start_thread at ??:0
@ 0x7fe83f77c61f clone at ??:0
W20260924 04:05:44.988379 15284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:44.988651 15284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:44.988694 15284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:44.992813 15284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:44.992883 15284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:44.992945 15284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:05:44.998420 15284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44007
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:44.999732 15284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:45.000933 15284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:45.011683 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:45.011685 15323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:45.012013 15326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:45.012166 15284 server_base.cc:1034] running on GCE node
I20260924 04:05:45.012605 15284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:45.013309 15284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:45.014477 15284 hybrid_clock.cc:648] HybridClock initialized: now 1790222745014435 us; error 53 us; skew 500 ppm
Sep 24 04:05:45 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:45.017279 15284 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:05:45.018187 15284 webserver.cc:460] Webserver started at http://127.0.98.1:44007/ using document root <none> and password file <none>
I20260924 04:05:45.018730 15284 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:45.018800 15284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:45.023324 15284 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:05:45.025669 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:45.027024 15284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:05:45.027140 15284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "095e15e59a70454aa39f29f9724ecceb"
format_stamp: "Formatted at 2026-09-24 04:05:42 on dist-test-slave-zcsk"
server_key: "ad4cb99ce7b3fa4f0837fba8cbc7eee1"
server_key_iv: "0e16aa62669ce3207d0a9579d3dc112c"
server_key_version: "encryptionkey@0"
I20260924 04:05:45.027690 15284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:45.052690 15284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:45.056020 15284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:45.056205 15284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:45.056869 15284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:45.058027 15340 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:05:45.062093 15284 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:05:45.062155 15284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 04:05:45.062206 15284 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:05:45.063876 15284 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:05:45.063961 15284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260924 04:05:45.064262 15340 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Bootstrap starting.
I20260924 04:05:45.074376 15284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36577
I20260924 04:05:45.074721 15447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36577 every 8 connection(s)
I20260924 04:05:45.075749 15284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:05:45.078408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 15284
I20260924 04:05:45.078842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40387
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=38349
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:45.084925 15340 log.cc:824] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:05:45.094512 15340 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:05:45.094940 15340 tablet_bootstrap.cc:492] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Bootstrap complete.
I20260924 04:05:45.096318 15340 ts_tablet_manager.cc:1397] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.016s
I20260924 04:05:45.099660 15340 raft_consensus.cc:348] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:45.100325 15340 raft_consensus.cc:729] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095e15e59a70454aa39f29f9724ecceb, State: Initialized, Role: FOLLOWER
I20260924 04:05:45.100878 15340 consensus_queue.cc:260] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:45.101960 15340 ts_tablet_manager.cc:1428] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
W20260924 04:05:45.110991 15450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:05:45.078300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:53915)
0924 04:05:45.079708 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.079770 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:45.080958 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:45.110239 (+ 29281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1154,"thread_start_us":183,"threads_started":1}
W20260924 04:05:45.111433 15448 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37025 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37025: Client connection negotiation failed: client connection to 127.0.98.62:37025: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:05:45.112097 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:45.111844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:59649)
0924 04:05:45.111986 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.112046 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 04:05:45.112772 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:45.112543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:37655)
0924 04:05:45.112685 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.112727 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:05:45.112931 15448 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:05:45.197497 15451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:45.197796 15451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:45.197846 15451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:45.201898 15451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:45.201964 15451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:45.202025 15451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:05:45.207336 15451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38349
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:45.208506 15451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:45.209519 15451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:45.219142 15458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:45.219187 15451 server_base.cc:1034] running on GCE node
W20260924 04:05:45.219146 15459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:45.219218 15461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:45.219799 15451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:45.220503 15451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:45.221691 15451 hybrid_clock.cc:648] HybridClock initialized: now 1790222745221661 us; error 46 us; skew 500 ppm
Sep 24 04:05:45 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:45.224558 15451 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:05:45.225528 15451 webserver.cc:460] Webserver started at http://127.0.98.2:38349/ using document root <none> and password file <none>
I20260924 04:05:45.226110 15451 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:45.226194 15451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:45.230332 15451 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:05:45.232613 15468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:45.233949 15451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:05:45.234067 15451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "59406b238da1406cb9d79aafc88a1912"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "4598b187adcb63e0b4b3d766fdf8528f"
server_key_iv: "57f830bce2a1c1e4913f318f9f6e2edb"
server_key_version: "encryptionkey@0"
I20260924 04:05:45.234644 15451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:45.253373 15451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:45.256704 15451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:45.256902 15451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:45.257522 15451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:45.258389 15451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:45.258437 15451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:45.258490 15451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:45.258518 15451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:45.267199 15451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40387
I20260924 04:05:45.267227 15581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40387 every 8 connection(s)
I20260924 04:05:45.268323 15451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:05:45.269018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 15451
I20260924 04:05:45.269392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38061
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=36603
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:05:45.272746 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:45.270650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:39567)
0924 04:05:45.271216 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.271368 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":124,"threads_started":1}
W20260924 04:05:45.273231 15582 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37025 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37025: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
I20260924 04:05:45.273912 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:45.273694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:59065)
0924 04:05:45.273806 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.273857 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:05:45.274499 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:45.274327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:52357)
0924 04:05:45.274415 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.274453 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:05:45.274647 15582 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:05:45.381764 15584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:05:45.382046 15584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:05:45.382090 15584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:05:45.386219 15584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:05:45.386286 15584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:05:45.386350 15584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:05:45.391755 15584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36603
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:05:45.392874 15584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:05:45.393882 15584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:05:45.403422 15590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:05:45.403460 15593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:45.403522 15584 server_base.cc:1034] running on GCE node
W20260924 04:05:45.403437 15591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:05:45.404140 15584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:05:45.404793 15584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:05:45.405982 15584 hybrid_clock.cc:648] HybridClock initialized: now 1790222745405960 us; error 35 us; skew 500 ppm
Sep 24 04:05:45 dist-test-slave-zcsk krb5kdc[14139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:05:45.408669 15584 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:05:45.409528 15584 webserver.cc:460] Webserver started at http://127.0.98.3:36603/ using document root <none> and password file <none>
I20260924 04:05:45.410054 15584 fs_manager.cc:359] Metadata directory not provided
I20260924 04:05:45.410120 15584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:05:45.413883 15584 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 04:05:45.415894 15600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:45.417120 15584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:05:45.417213 15584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "8eb72806452f45ab9ad9ed454df93f74"
format_stamp: "Formatted at 2026-09-24 04:05:43 on dist-test-slave-zcsk"
server_key: "a43306c119d1967e8185ea177503b7d2"
server_key_iv: "1cc2d8105aeac4f478bfeed4deec3d09"
server_key_version: "encryptionkey@0"
I20260924 04:05:45.417719 15584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:05:45.433157 15584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:05:45.436101 15584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:05:45.436270 15584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:05:45.436822 15584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:05:45.437652 15584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:05:45.437698 15584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:45.437779 15584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:05:45.437809 15584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:05:45.446468 15584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38061
I20260924 04:05:45.446511 15713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38061 every 8 connection(s)
I20260924 04:05:45.447479 15584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:05:45.451375 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:45.449564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:35389)
0924 04:05:45.450058 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.450172 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":142,"threads_started":1}
W20260924 04:05:45.451853 15714 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37025 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37025: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
I20260924 04:05:45.452507 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:45.452289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:55893)
0924 04:05:45.452403 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.452453 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 04:05:45.453081 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:45.452892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:42031)
0924 04:05:45.452988 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:45.453032 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:05:45.453230 15714 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:05:45.456003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 15584
I20260924 04:05:46.114094 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:46.113626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:52955)
0924 04:05:46.113920 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:46.113984 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
I20260924 04:05:46.275760 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:46.275293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:43017)
0924 04:05:46.275583 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:46.275651 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260924 04:05:46.454259 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:46.453795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:36091)
0924 04:05:46.454078 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:46.454154 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260924 04:05:46.583351 15721 raft_consensus.cc:482] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:46.583611 15721 raft_consensus.cc:504] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:46.584769 15721 leader_election.cc:288] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061), 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
Sep 24 04:05:46 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 04:05:46 dist-test-slave-zcsk krb5kdc[14139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 04:05:46.598174 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:46.585525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52653 (local address 127.0.98.2:40387)
0924 04:05:46.585901 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:46.585944 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:46.585980 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:05:46.586594 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:46.586599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:46.586877 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:46.587171 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:46.588321 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:46.589274 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:46.590516 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:46.591070 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:46.591229 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:46.594177 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:46.594182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:46.594194 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:46.594223 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:46.595813 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:46.596306 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:46.596311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:46.596315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:46.596423 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:46.596686 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:46.596689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:46.596690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:46.597085 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:46.597200 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:46.597742 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 04:05:46.597995 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":76,"threads_started":1}
I20260924 04:05:46.598173 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:46.585246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58645 (local address 127.0.98.3:38061)
0924 04:05:46.585631 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 04:05:46.585654 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:05:46.585693 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:05:46.586386 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:05:46.586392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:05:46.586736 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:05:46.586987 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:05:46.588326 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:46.589293 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:46.590475 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:05:46.591077 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:05:46.591235 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:46.593924 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:05:46.593930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:05:46.593944 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:05:46.593973 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:05:46.595670 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:46.596243 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:46.596250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:46.596255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:46.596367 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:05:46.596690 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:05:46.596693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:05:46.596695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:05:46.597086 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:05:46.597200 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:05:46.597746 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 04:05:46.597993 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260924 04:05:46.598191 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:46.585345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40387 (local address 127.0.98.1:52653)
0924 04:05:46.585755 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:46.585770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:46.585877 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:46.587156 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:46.587159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:46.587647 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:46.588168 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:46.588182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:46.589414 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:46.589418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:46.590372 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:46.590382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:46.591183 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:46.591186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:46.591350 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:46.592071 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:46.592087 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:46.593999 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:46.595950 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:46.595954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:46.595956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:46.596203 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:46.596520 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:46.596523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:46.596525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:46.596600 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:46.597192 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:46.597197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:46.597438 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 04:05:46.597594 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 04:05:46.598015 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":278,"spinlock_wait_cycles":26112,"thread_start_us":81,"threads_started":1}
I20260924 04:05:46.598214 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:46.585054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38061 (local address 127.0.98.1:58645)
0924 04:05:46.585204 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:46.585220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:05:46.585319 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:05:46.587087 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:05:46.587092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:05:46.587647 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:05:46.588168 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:46.588182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:46.589413 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:46.589415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:46.590364 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:05:46.590373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:05:46.591180 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:05:46.591184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:05:46.591349 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:05:46.592117 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:05:46.592128 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:05:46.593698 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:05:46.595793 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:46.595799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:46.595813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:46.596141 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:46.596452 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:05:46.596455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:05:46.596457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:05:46.596597 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:05:46.597190 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:05:46.597196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:05:46.597442 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 04:05:46.597594 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 04:05:46.597946 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":149,"spinlock_wait_cycles":28800}
W20260924 04:05:46.600688 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
W20260924 04:05:46.600939 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
I20260924 04:05:46.601015 15334 leader_election.cc:302] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 095e15e59a70454aa39f29f9724ecceb; no voters: 59406b238da1406cb9d79aafc88a1912, 8eb72806452f45ab9ad9ed454df93f74
I20260924 04:05:46.601230 15721 raft_consensus.cc:2726] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:05:47.115377 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:47.114948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:60641)
0924 04:05:47.115219 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:47.115284 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260924 04:05:47.276927 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:47.276476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:49307)
0924 04:05:47.276764 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:47.276821 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":78,"threads_started":1}
I20260924 04:05:47.455477 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:47.455017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:57895)
0924 04:05:47.455302 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:47.455368 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260924 04:05:48.116576 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:48.116149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:40357)
0924 04:05:48.116414 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:48.116477 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260924 04:05:48.278317 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:48.277866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:42815)
0924 04:05:48.278153 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:48.278219 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260924 04:05:48.456667 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:48.456220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:54273)
0924 04:05:48.456508 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:48.456569 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260924 04:05:48.770605 15731 raft_consensus.cc:482] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:48.770745 15731 raft_consensus.cc:504] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:48.771183 15731 leader_election.cc:288] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061), 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
W20260924 04:05:48.772220 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
W20260924 04:05:48.772378 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
I20260924 04:05:48.772431 15334 leader_election.cc:302] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 095e15e59a70454aa39f29f9724ecceb; no voters: 59406b238da1406cb9d79aafc88a1912, 8eb72806452f45ab9ad9ed454df93f74
I20260924 04:05:48.772596 15731 raft_consensus.cc:2726] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:05:49.117727 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:49.117267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:48767)
0924 04:05:49.117578 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:49.117631 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":134,"threads_started":1}
I20260924 04:05:49.279798 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:49.279301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:53951)
0924 04:05:49.279620 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:49.279689 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":82,"threads_started":1}
I20260924 04:05:49.458030 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:49.457565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:45169)
0924 04:05:49.457868 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:49.457936 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20260924 04:05:50.119141 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:50.118655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:56553)
0924 04:05:50.118981 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:50.119046 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
I20260924 04:05:50.281189 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:50.280716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:46499)
0924 04:05:50.281005 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:50.281070 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260924 04:05:50.459291 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:50.458828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:43985)
0924 04:05:50.459124 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:50.459190 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":81,"threads_started":1}
I20260924 04:05:51.120450 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:51.120001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:42699)
0924 04:05:51.120291 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:51.120355 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":73,"threads_started":1}
I20260924 04:05:51.282495 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:51.282046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:44387)
0924 04:05:51.282320 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:51.282392 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260924 04:05:51.396404 15740 raft_consensus.cc:482] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:51.396556 15740 raft_consensus.cc:504] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:51.397044 15740 leader_election.cc:288] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061), 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
W20260924 04:05:51.397887 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
W20260924 04:05:51.398031 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
I20260924 04:05:51.398085 15334 leader_election.cc:302] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 095e15e59a70454aa39f29f9724ecceb; no voters: 59406b238da1406cb9d79aafc88a1912, 8eb72806452f45ab9ad9ed454df93f74
I20260924 04:05:51.398227 15740 raft_consensus.cc:2726] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:05:51.460619 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:51.460169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:34425)
0924 04:05:51.460452 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:51.460519 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260924 04:05:52.121778 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:52.121255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:58133)
0924 04:05:52.121593 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:52.121677 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
I20260924 04:05:52.283743 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:52.283277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:53913)
0924 04:05:52.283582 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:52.283644 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":73,"threads_started":1}
I20260924 04:05:52.461818 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:52.461380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:47549)
0924 04:05:52.461660 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:52.461722 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
I20260924 04:05:53.123013 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:53.122574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:55151)
0924 04:05:53.122846 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:53.122912 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260924 04:05:53.285065 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:53.284614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:49173)
0924 04:05:53.284897 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:53.284964 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260924 04:05:53.463109 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:53.462640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:55439)
0924 04:05:53.462943 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:53.463010 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260924 04:05:54.124230 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:54.123758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:35443)
0924 04:05:54.124069 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:54.124132 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260924 04:05:54.286352 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:54.285863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:48049)
0924 04:05:54.286170 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:54.286238 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":81,"threads_started":1}
I20260924 04:05:54.446719 15750 raft_consensus.cc:482] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:54.446873 15750 raft_consensus.cc:504] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:54.447273 15750 leader_election.cc:288] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061), 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
W20260924 04:05:54.448115 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
W20260924 04:05:54.448269 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
I20260924 04:05:54.448331 15334 leader_election.cc:302] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 095e15e59a70454aa39f29f9724ecceb; no voters: 59406b238da1406cb9d79aafc88a1912, 8eb72806452f45ab9ad9ed454df93f74
I20260924 04:05:54.448477 15750 raft_consensus.cc:2726] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:05:54.464357 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:54.463864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:54809)
0924 04:05:54.464176 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:54.464239 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":79,"threads_started":1}
I20260924 04:05:55.125566 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:55.125177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:36741)
0924 04:05:55.125416 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:55.125472 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260924 04:05:55.287617 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:55.287169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:47433)
0924 04:05:55.287436 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:55.287505 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260924 04:05:55.465615 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:55.465155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:49549)
0924 04:05:55.465458 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:55.465519 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":115,"threads_started":1}
I20260924 04:05:56.126847 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:56.126383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:59685)
0924 04:05:56.126686 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:56.126751 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
I20260924 04:05:56.288801 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:56.288328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:52689)
0924 04:05:56.288637 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:56.288700 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
I20260924 04:05:56.466835 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:56.466413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:54289)
0924 04:05:56.466676 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:56.466738 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260924 04:05:57.128130 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:57.127621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:53367)
0924 04:05:57.127965 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:57.128028 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
I20260924 04:05:57.290160 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:57.289684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:39267)
0924 04:05:57.289995 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:57.290061 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":77,"threads_started":1}
I20260924 04:05:57.468150 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:57.467682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:56585)
0924 04:05:57.467975 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:57.468048 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260924 04:05:58.129459 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:58.129028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:49765)
0924 04:05:58.129301 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:58.129365 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260924 04:05:58.291518 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:58.291002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:54419)
0924 04:05:58.291350 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:58.291418 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260924 04:05:58.368357 15763 raft_consensus.cc:482] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:05:58.368511 15763 raft_consensus.cc:504] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb72806452f45ab9ad9ed454df93f74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38061 } } peers { permanent_uuid: "59406b238da1406cb9d79aafc88a1912" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40387 } } peers { permanent_uuid: "095e15e59a70454aa39f29f9724ecceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36577 } }
I20260924 04:05:58.368933 15763 leader_election.cc:288] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061), 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387)
W20260924 04:05:58.369846 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59406b238da1406cb9d79aafc88a1912 (127.0.98.2:40387): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
W20260924 04:05:58.369989 15334 leader_election.cc:341] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eb72806452f45ab9ad9ed454df93f74 (127.0.98.3:38061): Not found: Tablet not found: 36bb5a8b1b37420e9621e5e070d3a388
I20260924 04:05:58.370043 15334 leader_election.cc:302] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 095e15e59a70454aa39f29f9724ecceb; no voters: 59406b238da1406cb9d79aafc88a1912, 8eb72806452f45ab9ad9ed454df93f74
I20260924 04:05:58.370213 15763 raft_consensus.cc:2726] T 36bb5a8b1b37420e9621e5e070d3a388 P 095e15e59a70454aa39f29f9724ecceb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:05:58.469476 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:58.469008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:55057)
0924 04:05:58.469311 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:58.469379 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260924 04:05:59.130733 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:59.130244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:48521)
0924 04:05:59.130554 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:59.130631 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
I20260924 04:05:59.292874 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:59.292396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:36899)
0924 04:05:59.292714 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:59.292778 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":131,"threads_started":1}
I20260924 04:05:59.470881 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:05:59.470356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:56211)
0924 04:05:59.470705 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:05:59.470782 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":100,"threads_started":1}
I20260924 04:06:00.132160 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:00.131631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:44085)
0924 04:06:00.131980 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:00.132053 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":120,"threads_started":1}
I20260924 04:06:00.294296 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:00.293851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:45003)
0924 04:06:00.294138 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:00.294199 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260924 04:06:00.458415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 15284
I20260924 04:06:00.463560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 15451
I20260924 04:06:00.468196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 15584
2026-09-24T04:06:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18989 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:06:00 dist-test-slave-zcsk krb5kdc[15772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:06:00 dist-test-slave-zcsk krb5kdc[15772](info): set up 2 sockets
Sep 24 04:06:00 dist-test-slave-zcsk krb5kdc[15772](info): commencing operation
krb5kdc: starting...
W20260924 04:06:01.423856 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:06:01 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:06:01.559544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:06:01.671463 15785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:01.671792 15785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:01.671852 15785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:01.676381 15785 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:06:01.676458 15785 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:06:01.676481 15785 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:06:01.676506 15785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:06:01.676530 15785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:01.676556 15785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:01.676586 15785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:06:01.676616 15785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:06:01.682269 15785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:01.683415 15785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:01.684551 15785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:01.693315 15791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:01.693353 15785 server_base.cc:1034] running on GCE node
W20260924 04:06:01.693317 15790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:01.693315 15793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:01.694183 15785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:01.695277 15785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:01.696435 15785 hybrid_clock.cc:648] HybridClock initialized: now 1790222761696415 us; error 36 us; skew 500 ppm
Sep 24 04:06:01 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:01.699167 15785 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:06:01.700140 15785 webserver.cc:460] Webserver started at http://127.0.98.62:33357/ using document root <none> and password file <none>
I20260924 04:06:01.700680 15785 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:01.700748 15785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:01.700965 15785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:06:01.703063 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "a576b50844ec4ce7a1794a002dc7558e"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "5dcaaeb6fbee1dc7b2e8b7d168421a1c"
server_key_iv: "c34dffeef51bfba71d6cccef7e6d299d"
server_key_version: "encryptionkey@0"
I20260924 04:06:01.703655 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "a576b50844ec4ce7a1794a002dc7558e"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "5dcaaeb6fbee1dc7b2e8b7d168421a1c"
server_key_iv: "c34dffeef51bfba71d6cccef7e6d299d"
server_key_version: "encryptionkey@0"
I20260924 04:06:01.707657 15785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 04:06:01.710345 15800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:01.711681 15785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:06:01.711814 15785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "a576b50844ec4ce7a1794a002dc7558e"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "5dcaaeb6fbee1dc7b2e8b7d168421a1c"
server_key_iv: "c34dffeef51bfba71d6cccef7e6d299d"
server_key_version: "encryptionkey@0"
I20260924 04:06:01.711969 15785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:01.724437 15785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:01.727892 15785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:01.728134 15785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:01.735013 15785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38667
I20260924 04:06:01.735280 15852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38667 every 8 connection(s)
I20260924 04:06:01.736174 15785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:06:01.738911 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:06:01.744681 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Bootstrap starting.
I20260924 04:06:01.745977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 15785
I20260924 04:06:01.746106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:06:01.746392 392 external_mini_cluster.cc:1428] Setting key 77e0849cd1c437ed98c29dfb42683036
I20260924 04:06:01.747721 15853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Neither blocks nor log segments found. Creating new log.
I20260924 04:06:01.748664 15853 log.cc:824] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:01.750854 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: No bootstrap required, opened a new log
I20260924 04:06:01.754894 15853 raft_consensus.cc:348] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:01.755134 15853 raft_consensus.cc:374] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:06:01.755192 15853 raft_consensus.cc:729] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a576b50844ec4ce7a1794a002dc7558e, State: Initialized, Role: FOLLOWER
I20260924 04:06:01.755610 15853 consensus_queue.cc:260] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
Sep 24 04:06:01 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:06:01.755735 15853 raft_consensus.cc:388] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:06:01.755801 15853 raft_consensus.cc:482] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:06:01.755894 15853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:06:01.757054 15853 raft_consensus.cc:504] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:01.757442 15853 leader_election.cc:302] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a576b50844ec4ce7a1794a002dc7558e; no voters:
I20260924 04:06:01.757835 15853 leader_election.cc:288] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:06:01.757920 15858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:06:01.758142 15858 raft_consensus.cc:686] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 1 LEADER]: Becoming Leader. State: Replica: a576b50844ec4ce7a1794a002dc7558e, State: Running, Role: LEADER
I20260924 04:06:01.758533 15858 consensus_queue.cc:237] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:01.759408 15853 sys_catalog.cc:564] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:06:01.760336 15859 sys_catalog.cc:455] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a576b50844ec4ce7a1794a002dc7558e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } } }
I20260924 04:06:01.760468 15859 sys_catalog.cc:458] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: This master's current role is: LEADER
I20260924 04:06:01.760871 15867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:06:01.761780 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a576b50844ec4ce7a1794a002dc7558e. Latest consensus state: current_term: 1 leader_uuid: "a576b50844ec4ce7a1794a002dc7558e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } } }
I20260924 04:06:01.761970 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: This master's current role is: LEADER
I20260924 04:06:01.762370 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:01.747583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.0.98.62:38667)
0924 04:06:01.748068 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:01.748088 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:01.748132 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:06:01.749000 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:01.749027 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:01.749381 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:01.749761 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:01.750469 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:01.751963 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:01.752852 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:01.753573 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:01.753729 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:01.756324 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:01.756331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:01.756349 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:01.756381 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:01.758494 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:01.759028 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:01.759034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:01.759041 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:01.759188 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:01.759530 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:01.759533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:01.759535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:01.760105 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:01.760309 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:01.761055 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 04:06:01.761358 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":136,"threads_started":1}
I20260924 04:06:01.764529 15867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:06:01.770220 15867 catalog_manager.cc:1347] Generated new cluster ID: 6abd2432b0344acfb44a82e5fde11a50
I20260924 04:06:01.770328 15867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:06:01.788627 15867 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:06:01.789654 15867 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:06:01.796285 15867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Generated new TSK 0
I20260924 04:06:01.797015 15867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:06:01.830924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38667
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:06:01.956372 15879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:01.956666 15879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:01.956732 15879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:01.961107 15879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:01.961186 15879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:01.961266 15879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:06:01.966791 15879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:01.967942 15879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:01.968986 15879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:01.978914 15884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:01.978901 15887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:01.978901 15885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:01.979125 15879 server_base.cc:1034] running on GCE node
I20260924 04:06:01.979607 15879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:01.980350 15879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:01.981526 15879 hybrid_clock.cc:648] HybridClock initialized: now 1790222761981502 us; error 48 us; skew 500 ppm
Sep 24 04:06:01 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:01.984352 15879 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:06:01.985289 15879 webserver.cc:460] Webserver started at http://127.0.98.1:39269/ using document root <none> and password file <none>
I20260924 04:06:01.985832 15879 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:01.985899 15879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:01.986117 15879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:06:01.988206 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "db685c3bb2334df099ed99538d7fd8be"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "22c8c000fccedec28805dd5afb00635b"
server_key_iv: "481c9f5ff1af64ebcdceb72dc7a915a0"
server_key_version: "encryptionkey@0"
I20260924 04:06:01.988746 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "db685c3bb2334df099ed99538d7fd8be"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "22c8c000fccedec28805dd5afb00635b"
server_key_iv: "481c9f5ff1af64ebcdceb72dc7a915a0"
server_key_version: "encryptionkey@0"
I20260924 04:06:01.992522 15879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:06:01.995050 15894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:01.996469 15879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:06:01.996596 15879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "db685c3bb2334df099ed99538d7fd8be"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "22c8c000fccedec28805dd5afb00635b"
server_key_iv: "481c9f5ff1af64ebcdceb72dc7a915a0"
server_key_version: "encryptionkey@0"
I20260924 04:06:01.996699 15879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:02.014523 15879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:02.018899 15879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:02.019109 15879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:02.019873 15879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:02.020861 15879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:06:02.020921 15879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:02.020988 15879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:06:02.021015 15879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:02.029791 15879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39039
I20260924 04:06:02.029831 16007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39039 every 8 connection(s)
I20260924 04:06:02.030736 15879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:06:02.038295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 15879
I20260924 04:06:02.038447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:06:02.038738 392 external_mini_cluster.cc:1428] Setting key 08e2ea2ad6e4f4e8a22ff770d12a4971
Sep 24 04:06:02 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:06:02.048529 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.032485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41453 (local address 127.0.98.62:38667)
0924 04:06:02.032618 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.032622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.033756 (+ 1134us) server_negotiation.cc:407] Connection header received
0924 04:06:02.034637 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.034640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.034689 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.034765 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:02.036206 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.036749 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.038035 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.038363 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.038476 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.044460 (+ 5984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:02.044466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:02.044469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:02.044525 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:02.046110 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:02.046677 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:02.046682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:02.046684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:02.046773 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:02.047140 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:02.047143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:02.047145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:02.047391 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:02.047511 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.048186 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.048338 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:06:02.049458 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.032892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:41453)
0924 04:06:02.033597 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.033644 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:02.034447 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:02.035031 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:02.035039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:02.035424 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:02.036029 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.036046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.036918 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.036922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.037897 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.037908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.038557 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.038562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.038724 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.042576 (+ 3852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:02.042602 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:02.044246 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:02.046252 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:02.046258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:02.046274 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:02.046570 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:02.046859 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:02.046862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:02.046863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:02.047045 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:02.047538 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:02.047544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:02.047834 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 04:06:02.048125 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 04:06:02.048497 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:06:02.050730 16008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38667
I20260924 04:06:02.051015 16008 heartbeater.cc:464] Registering TS with master...
I20260924 04:06:02.051513 16008 heartbeater.cc:511] Master 127.0.98.62:38667 requested a full tablet report, sending...
I20260924 04:06:02.053292 15817 ts_manager.cc:194] Registered new tserver with Master: db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039)
I20260924 04:06:02.054474 15817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:06:02.068121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38667
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:06:02.179559 16013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:02.179983 16013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:02.180061 16013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:02.184314 16013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:02.184381 16013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:02.184443 16013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:06:02.189826 16013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:02.190855 16013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:02.191802 16013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:06:02.201638 16013 server_base.cc:1034] running on GCE node
W20260924 04:06:02.201665 16019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:02.201659 16018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:02.201617 16021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:02.202306 16013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:02.203083 16013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:02.204290 16013 hybrid_clock.cc:648] HybridClock initialized: now 1790222762204264 us; error 37 us; skew 500 ppm
Sep 24 04:06:02 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:02.207021 16013 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:06:02.207962 16013 webserver.cc:460] Webserver started at http://127.0.98.2:34367/ using document root <none> and password file <none>
I20260924 04:06:02.208525 16013 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:02.208587 16013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:02.208750 16013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:06:02.210793 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa80f2c31eea49a9a84b277d29502622"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "dc8d5cd40ca5632e1d437b727d47a41f"
server_key_iv: "b9a88ddb12e655ce92fdc512724f82d9"
server_key_version: "encryptionkey@0"
I20260924 04:06:02.211287 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa80f2c31eea49a9a84b277d29502622"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "dc8d5cd40ca5632e1d437b727d47a41f"
server_key_iv: "b9a88ddb12e655ce92fdc512724f82d9"
server_key_version: "encryptionkey@0"
I20260924 04:06:02.215040 16013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:06:02.217666 16028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:02.218963 16013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:06:02.219077 16013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "fa80f2c31eea49a9a84b277d29502622"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "dc8d5cd40ca5632e1d437b727d47a41f"
server_key_iv: "b9a88ddb12e655ce92fdc512724f82d9"
server_key_version: "encryptionkey@0"
I20260924 04:06:02.219205 16013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:02.230765 16013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:02.233765 16013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:02.233948 16013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:02.234555 16013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:02.235427 16013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:06:02.235479 16013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:02.235522 16013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:06:02.235550 16013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:02.244251 16013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40053
I20260924 04:06:02.244325 16141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40053 every 8 connection(s)
I20260924 04:06:02.245167 16013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:06:02 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:06:02.254602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 16013
I20260924 04:06:02.254720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:06:02.254957 392 external_mini_cluster.cc:1428] Setting key f6a776fe268f490437695158576d8e35
I20260924 04:06:02.258919 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.247015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55985 (local address 127.0.98.62:38667)
0924 04:06:02.247160 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.247164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.248049 (+ 885us) server_negotiation.cc:407] Connection header received
0924 04:06:02.249274 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.249277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.249326 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.249403 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:02.250554 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.251117 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.252286 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.252534 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.252622 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.254972 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:02.254975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:02.254977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:02.255002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:02.256668 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:02.257213 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:02.257218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:02.257220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:02.257312 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:02.257677 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:02.257679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:02.257681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:02.257891 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:02.257979 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.258606 (+ 627us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.258740 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:06:02.260078 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.247400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:55985)
0924 04:06:02.247851 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.247895 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:02.249067 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:02.249553 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:02.249561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:02.249918 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:02.250426 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.250439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.251216 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.251218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.252156 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.252166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.252655 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.252657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.252791 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.253360 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:02.253376 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:02.254814 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:02.256794 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:02.256804 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:02.256820 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:02.257113 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:02.257432 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:02.257435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:02.257436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:02.257594 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:02.258028 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:02.258036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:02.258343 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 04:06:02.258537 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 04:06:02.258853 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260924 04:06:02.261718 16142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38667
I20260924 04:06:02.261982 16142 heartbeater.cc:464] Registering TS with master...
I20260924 04:06:02.262530 16142 heartbeater.cc:511] Master 127.0.98.62:38667 requested a full tablet report, sending...
I20260924 04:06:02.263700 15817 ts_manager.cc:194] Registered new tserver with Master: fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053)
I20260924 04:06:02.264340 15817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55985
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:06:02.283814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38667
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:06:02.394583 16147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:02.394883 16147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:02.394948 16147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:02.399121 16147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:02.399194 16147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:02.399279 16147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:06:02.404786 16147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:02.405925 16147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:02.407042 16147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:02.416576 16153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:02.416574 16155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:02.416574 16152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:02.416994 16147 server_base.cc:1034] running on GCE node
I20260924 04:06:02.417382 16147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:02.418011 16147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:02.419124 16147 hybrid_clock.cc:648] HybridClock initialized: now 1790222762419102 us; error 36 us; skew 500 ppm
Sep 24 04:06:02 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:02.421988 16147 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:06:02.422894 16147 webserver.cc:460] Webserver started at http://127.0.98.3:43487/ using document root <none> and password file <none>
I20260924 04:06:02.423453 16147 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:02.423552 16147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:02.423763 16147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:06:02.426023 16147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "1ff5879648834db5b6a1a834ead5a00f"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "43945f44901297a764abf5441cd08ecc"
server_key_iv: "64ffa037b37f25aa3724e91e276d16a9"
server_key_version: "encryptionkey@0"
I20260924 04:06:02.426656 16147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1ff5879648834db5b6a1a834ead5a00f"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "43945f44901297a764abf5441cd08ecc"
server_key_iv: "64ffa037b37f25aa3724e91e276d16a9"
server_key_version: "encryptionkey@0"
I20260924 04:06:02.430485 16147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:06:02.433266 16162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:02.434551 16147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:06:02.434672 16147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "1ff5879648834db5b6a1a834ead5a00f"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "43945f44901297a764abf5441cd08ecc"
server_key_iv: "64ffa037b37f25aa3724e91e276d16a9"
server_key_version: "encryptionkey@0"
I20260924 04:06:02.434783 16147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:02.445393 16147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:02.448653 16147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:02.448843 16147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:02.449440 16147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:02.450292 16147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:06:02.450340 16147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:02.450392 16147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:06:02.450423 16147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:02.458705 16147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37963
I20260924 04:06:02.458727 16275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37963 every 8 connection(s)
I20260924 04:06:02.459623 16147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:06:02.459925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 16147
I20260924 04:06:02.460052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:06:02.460364 392 external_mini_cluster.cc:1428] Setting key 69be756eba38bd8d4e81df6e36faa4e6
Sep 24 04:06:02 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:06:02.473294 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.461686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60723 (local address 127.0.98.62:38667)
0924 04:06:02.461830 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.461835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.463035 (+ 1200us) server_negotiation.cc:407] Connection header received
0924 04:06:02.463981 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.463985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.464060 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.464134 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:02.465345 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.465916 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.467044 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.467282 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.467373 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.469871 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:02.469874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:02.469877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:02.469904 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:02.471175 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:02.471674 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:02.471677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:02.471679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:02.471767 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:02.472128 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:02.472132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:02.472133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:02.472346 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:02.472448 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.472995 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.473126 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:06:02.474165 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.462169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:60723)
0924 04:06:02.462877 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.462931 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:02.463731 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:02.464293 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:02.464300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:02.464697 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:02.465191 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.465204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.466033 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.466036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.466921 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.466930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.467411 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.467413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.467574 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.468195 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:02.468212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:02.469683 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:02.471283 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:02.471288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:02.471302 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:02.471569 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:02.471842 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:02.471844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:02.471845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:02.472021 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:02.472439 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:02.472461 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:02.472715 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 04:06:02.472889 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:06:02.473237 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":111,"threads_started":1}
I20260924 04:06:02.475150 16276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38667
I20260924 04:06:02.475389 16276 heartbeater.cc:464] Registering TS with master...
I20260924 04:06:02.475884 16276 heartbeater.cc:511] Master 127.0.98.62:38667 requested a full tablet report, sending...
I20260924 04:06:02.476941 15817 ts_manager.cc:194] Registered new tserver with Master: 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
I20260924 04:06:02.477499 15817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60723
I20260924 04:06:02.486459 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:06:02.495873 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.488489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.0.98.62:38667)
0924 04:06:02.488638 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.488643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.488732 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:06:02.488874 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.488877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.488921 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.488988 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:02.489665 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.490225 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.491064 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.491272 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.491362 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.492670 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:02.492673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:02.492675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:02.492700 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:02.494048 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:02.494515 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:02.494519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:02.494521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:02.494622 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:02.494903 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:02.494907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:02.494909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:02.495153 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:02.495284 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.495546 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.495687 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:06:02.498670 15817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:06:02.501751 15817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:06:02.516052 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.511304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.0.98.3:37963)
0924 04:06:02.511719 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.511731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.511751 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:06:02.511825 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.511830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.512057 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.512203 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:02.512416 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.513410 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.514716 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.515450 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.515572 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.515728 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.515778 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.515865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":134,"threads_started":1}
I20260924 04:06:02.516026 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.510082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40053 (local address 127.0.0.1:37812)
0924 04:06:02.510927 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.510940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:02.511028 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:02.511566 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:02.511571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:02.511594 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:02.511749 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.511756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.512982 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.512985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.514578 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.514586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.515379 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.515383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.515485 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.515523 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:06:02.515626 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:06:02.515824 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":206,"thread_start_us":86,"threads_started":1}
I20260924 04:06:02.516026 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.510748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37963 (local address 127.0.0.1:39000)
0924 04:06:02.511398 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.511409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:02.511502 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:02.512191 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:02.512194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:02.512211 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:02.512296 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.512301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.513532 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.513536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.514590 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.514599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.515573 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.515577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.515670 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.515689 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:06:02.515740 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:06:02.515867 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":66,"thread_start_us":104,"threads_started":1}
I20260924 04:06:02.516234 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.510172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.0.98.1:39039)
0924 04:06:02.510426 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.510444 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.511184 (+ 740us) server_negotiation.cc:407] Connection header received
0924 04:06:02.511256 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.511260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.511412 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.511543 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:02.511890 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.513090 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.514552 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.515372 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.515800 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.515956 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.516005 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.516095 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260924 04:06:02.516263 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.510019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39039 (local address 127.0.0.1:39544)
0924 04:06:02.510790 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.510843 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:02.510989 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:02.511707 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:02.511710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:02.511724 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:02.511790 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.511796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.513214 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.513217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.514423 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.514433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.515534 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.515538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.515633 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.515654 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:06:02.515701 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:06:02.516069 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":95,"thread_start_us":77,"threads_started":1}
I20260924 04:06:02.516721 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.510651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.0.98.2:40053)
0924 04:06:02.511068 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.511080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.511103 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:06:02.511179 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.511184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.511395 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.511934 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:02.512072 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.512869 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.514678 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.515242 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.515336 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.515492 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.516504 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.516584 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":107,"threads_started":1}
I20260924 04:06:02.519138 16076 tablet_service.cc:1467] Processing CreateTablet for tablet e201a98af6764829b8287eb93256d88f (DEFAULT_TABLE table=test-table [id=bbfbcde17aec47e892266f5fb77998ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:06:02.519181 15942 tablet_service.cc:1467] Processing CreateTablet for tablet e201a98af6764829b8287eb93256d88f (DEFAULT_TABLE table=test-table [id=bbfbcde17aec47e892266f5fb77998ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:06:02.519181 16210 tablet_service.cc:1467] Processing CreateTablet for tablet e201a98af6764829b8287eb93256d88f (DEFAULT_TABLE table=test-table [id=bbfbcde17aec47e892266f5fb77998ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:06:02.520213 16076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e201a98af6764829b8287eb93256d88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:06:02.520218 15942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e201a98af6764829b8287eb93256d88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:06:02.520223 16210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e201a98af6764829b8287eb93256d88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:06:02.527634 16295 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Bootstrap starting.
I20260924 04:06:02.528242 16294 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: Bootstrap starting.
I20260924 04:06:02.529703 16295 tablet_bootstrap.cc:654] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Neither blocks nor log segments found. Creating new log.
I20260924 04:06:02.530035 16294 tablet_bootstrap.cc:654] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: Neither blocks nor log segments found. Creating new log.
I20260924 04:06:02.530798 16295 log.cc:824] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:02.531018 16294 log.cc:824] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:02.531718 16293 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: Bootstrap starting.
I20260924 04:06:02.532931 16295 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: No bootstrap required, opened a new log
I20260924 04:06:02.532931 16294 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: No bootstrap required, opened a new log
I20260924 04:06:02.533138 16294 ts_tablet_manager.cc:1397] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:06:02.533138 16295 ts_tablet_manager.cc:1397] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:06:02.534217 16293 tablet_bootstrap.cc:654] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: Neither blocks nor log segments found. Creating new log.
I20260924 04:06:02.534955 16293 log.cc:824] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:02.536899 16293 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: No bootstrap required, opened a new log
I20260924 04:06:02.537092 16293 ts_tablet_manager.cc:1397] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:06:02.537307 16295 raft_consensus.cc:348] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.537307 16294 raft_consensus.cc:348] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.537607 16295 raft_consensus.cc:374] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:06:02.537607 16294 raft_consensus.cc:374] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:06:02.537667 16294 raft_consensus.cc:729] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff5879648834db5b6a1a834ead5a00f, State: Initialized, Role: FOLLOWER
I20260924 04:06:02.537704 16295 raft_consensus.cc:729] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db685c3bb2334df099ed99538d7fd8be, State: Initialized, Role: FOLLOWER
I20260924 04:06:02.538223 16295 consensus_queue.cc:260] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.538223 16294 consensus_queue.cc:260] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.538969 16276 heartbeater.cc:503] Master 127.0.98.62:38667 was elected leader, sending a full tablet report...
I20260924 04:06:02.539289 16008 heartbeater.cc:503] Master 127.0.98.62:38667 was elected leader, sending a full tablet report...
I20260924 04:06:02.540381 16294 ts_tablet_manager.cc:1428] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260924 04:06:02.540604 16295 ts_tablet_manager.cc:1428] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:06:02.541729 16293 raft_consensus.cc:348] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.541982 16293 raft_consensus.cc:374] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:06:02.542040 16293 raft_consensus.cc:729] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa80f2c31eea49a9a84b277d29502622, State: Initialized, Role: FOLLOWER
I20260924 04:06:02.542461 16293 consensus_queue.cc:260] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.543118 16142 heartbeater.cc:503] Master 127.0.98.62:38667 was elected leader, sending a full tablet report...
I20260924 04:06:02.543439 16293 ts_tablet_manager.cc:1428] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:06:02.630739 16300 raft_consensus.cc:482] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:06:02.630995 16300 raft_consensus.cc:504] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.632316 16300 leader_election.cc:288] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
I20260924 04:06:02.636333 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.632645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51957 (local address 127.0.98.2:40053)
0924 04:06:02.632784 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.632789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.632923 (+ 134us) server_negotiation.cc:407] Connection header received
0924 04:06:02.633110 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.633114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.633168 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.633246 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:02.633539 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.634123 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.635309 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.635755 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.635834 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.635887 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.636122 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.636200 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:06:02.636366 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.633075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52225 (local address 127.0.98.3:37963)
0924 04:06:02.633198 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.633201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.633266 (+ 65us) server_negotiation.cc:407] Connection header received
0924 04:06:02.633389 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.633392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.633438 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.633518 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:02.633820 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.634355 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.635381 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.635846 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.636000 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.636060 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.636193 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.636259 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:06:02.636354 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.632541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40053 (local address 127.0.98.1:51957)
0924 04:06:02.632838 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.632851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:02.632988 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:02.633278 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:02.633281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:02.633305 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:02.633433 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.633440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.634235 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.634239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.635186 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.635196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.635873 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.635876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.636000 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.636026 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:06:02.636097 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:06:02.636166 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":23,"thread_start_us":104,"threads_started":1}
I20260924 04:06:02.636360 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.632929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37963 (local address 127.0.98.1:52225)
0924 04:06:02.633205 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.633218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:02.633298 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:02.633535 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:02.633538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:02.633555 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:02.633667 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.633673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.634461 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.634464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.635266 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:02.635277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.635995 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.635998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:02.636079 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.636103 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:06:02.636173 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:06:02.636235 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":22,"thread_start_us":100,"threads_started":1}
I20260924 04:06:02.637107 16096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e201a98af6764829b8287eb93256d88f" candidate_uuid: "db685c3bb2334df099ed99538d7fd8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa80f2c31eea49a9a84b277d29502622" is_pre_election: true
I20260924 04:06:02.637104 16230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e201a98af6764829b8287eb93256d88f" candidate_uuid: "db685c3bb2334df099ed99538d7fd8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff5879648834db5b6a1a834ead5a00f" is_pre_election: true
I20260924 04:06:02.637382 16230 raft_consensus.cc:2454] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db685c3bb2334df099ed99538d7fd8be in term 0.
I20260924 04:06:02.637382 16096 raft_consensus.cc:2454] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db685c3bb2334df099ed99538d7fd8be in term 0.
I20260924 04:06:02.637820 15897 leader_election.cc:302] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff5879648834db5b6a1a834ead5a00f, db685c3bb2334df099ed99538d7fd8be; no voters:
I20260924 04:06:02.638083 16300 raft_consensus.cc:2781] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:06:02.638159 16300 raft_consensus.cc:482] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:06:02.638191 16300 raft_consensus.cc:3037] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:06:02.639261 16300 raft_consensus.cc:504] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.639598 16300 leader_election.cc:288] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 1 election: Requested vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
I20260924 04:06:02.640012 16096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e201a98af6764829b8287eb93256d88f" candidate_uuid: "db685c3bb2334df099ed99538d7fd8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa80f2c31eea49a9a84b277d29502622"
I20260924 04:06:02.640077 16230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e201a98af6764829b8287eb93256d88f" candidate_uuid: "db685c3bb2334df099ed99538d7fd8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff5879648834db5b6a1a834ead5a00f"
I20260924 04:06:02.640142 16096 raft_consensus.cc:3037] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:06:02.640203 16230 raft_consensus.cc:3037] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:06:02.641175 16230 raft_consensus.cc:2454] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db685c3bb2334df099ed99538d7fd8be in term 1.
I20260924 04:06:02.641187 16096 raft_consensus.cc:2454] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db685c3bb2334df099ed99538d7fd8be in term 1.
I20260924 04:06:02.641453 15897 leader_election.cc:302] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db685c3bb2334df099ed99538d7fd8be, fa80f2c31eea49a9a84b277d29502622; no voters:
I20260924 04:06:02.641642 16300 raft_consensus.cc:2781] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:06:02.641865 16300 raft_consensus.cc:686] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 1 LEADER]: Becoming Leader. State: Replica: db685c3bb2334df099ed99538d7fd8be, State: Running, Role: LEADER
I20260924 04:06:02.642225 16300 consensus_queue.cc:237] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:02.645562 15817 catalog_manager.cc:5495] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be reported cstate change: term changed from 0 to 1, leader changed from <none> to db685c3bb2334df099ed99538d7fd8be (127.0.98.1). New cstate: current_term: 1 leader_uuid: "db685c3bb2334df099ed99538d7fd8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } health_report { overall_health: UNKNOWN } } }
I20260924 04:06:02.669548 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.666086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.0.98.1:39039)
0924 04:06:02.666209 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:02.666214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:02.666228 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:06:02.666332 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:02.666335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:02.666394 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:02.666487 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:06:02.666715 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.667228 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.668139 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:02.668360 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:02.668450 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:02.668743 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:06:02.669173 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:06:02.669282 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:02.669347 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 04:06:02.669408 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:06:02.675412 16096 raft_consensus.cc:1261] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 1 FOLLOWER]: Refusing update from remote peer db685c3bb2334df099ed99538d7fd8be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:06:02.675412 16230 raft_consensus.cc:1261] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Refusing update from remote peer db685c3bb2334df099ed99538d7fd8be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:06:02.676154 16300 consensus_queue.cc:1035] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:06:02.676396 16304 consensus_queue.cc:1035] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:06:02.684167 16308 mvcc.cc:205] Tried to move back new op lower bound from 7332752435911446528 to 7332752435785289728. Current Snapshot: MvccSnapshot[applied={T|T < 7332752435911446528}]
I20260924 04:06:02.685257 16309 mvcc.cc:205] Tried to move back new op lower bound from 7332752435911446528 to 7332752435785289728. Current Snapshot: MvccSnapshot[applied={T|T < 7332752435911446528}]
I20260924 04:06:02.692224 15817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:06:02.692431 15817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:06:02.695019 15817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Sending DeleteTablet for 3 replicas of tablet e201a98af6764829b8287eb93256d88f
I20260924 04:06:02.695823 15942 tablet_service.cc:1514] Processing DeleteTablet for tablet e201a98af6764829b8287eb93256d88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:06:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39544
I20260924 04:06:02.695888 16076 tablet_service.cc:1514] Processing DeleteTablet for tablet e201a98af6764829b8287eb93256d88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:06:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37812
I20260924 04:06:02.696249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 15879
I20260924 04:06:02.696415 16210 tablet_service.cc:1514] Processing DeleteTablet for tablet e201a98af6764829b8287eb93256d88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:06:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39000
I20260924 04:06:02.697009 16318 tablet_replica.cc:330] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: stopping tablet replica
I20260924 04:06:02.697342 16318 raft_consensus.cc:2229] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:06:02.697516 16319 tablet_replica.cc:330] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: stopping tablet replica
I20260924 04:06:02.697785 16318 raft_consensus.cc:2258] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:06:02.697817 16319 raft_consensus.cc:2229] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:06:02.698096 16319 raft_consensus.cc:2258] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:06:02.699170 16318 ts_tablet_manager.cc:1905] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:06:02.699493 16319 ts_tablet_manager.cc:1905] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:06:02.701591 16318 ts_tablet_manager.cc:1918] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:06:02.701687 16318 log.cc:1196] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/e201a98af6764829b8287eb93256d88f
I20260924 04:06:02.701972 16318 ts_tablet_manager.cc:1939] T e201a98af6764829b8287eb93256d88f P fa80f2c31eea49a9a84b277d29502622: Deleting consensus metadata
I20260924 04:06:02.702121 16319 ts_tablet_manager.cc:1918] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:06:02.702219 16319 log.cc:1196] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/e201a98af6764829b8287eb93256d88f
I20260924 04:06:02.702565 16319 ts_tablet_manager.cc:1939] T e201a98af6764829b8287eb93256d88f P 1ff5879648834db5b6a1a834ead5a00f: Deleting consensus metadata
I20260924 04:06:02.703375 15803 catalog_manager.cc:4841] TS fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053): tablet e201a98af6764829b8287eb93256d88f (table test-table [id=bbfbcde17aec47e892266f5fb77998ab]) successfully deleted
W20260924 04:06:02.703536 15801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39039 (error 108)
I20260924 04:06:02.703747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 16013
I20260924 04:06:02.704573 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:02.704253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39039 (local address 127.0.0.1:39566)
0924 04:06:02.704423 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:02.704490 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 04:06:02.704710 15801 catalog_manager.cc:4584] TS db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e201a98af6764829b8287eb93256d88f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39039: connect: Connection refused (error 111)
I20260924 04:06:02.705646 15803 catalog_manager.cc:4841] TS 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963): tablet e201a98af6764829b8287eb93256d88f (table test-table [id=bbfbcde17aec47e892266f5fb77998ab]) successfully deleted
I20260924 04:06:02.709175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 16147
I20260924 04:06:02.713871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 15785
I20260924 04:06:22.719673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38667
--webserver_interface=127.0.98.62
--webserver_port=33357
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:06:22.832793 16321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:22.833070 16321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:22.833110 16321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:22.837255 16321 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:06:22.837322 16321 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:06:22.837337 16321 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:06:22.837349 16321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:06:22.837368 16321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:22.837384 16321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:22.837399 16321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:06:22.837419 16321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:06:22.842875 16321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:22.844019 16321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:22.845064 16321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:22.859177 16327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:22.859196 16328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:22.859264 16321 server_base.cc:1034] running on GCE node
W20260924 04:06:22.859174 16330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:22.860118 16321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:22.861435 16321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:22.862602 16321 hybrid_clock.cc:648] HybridClock initialized: now 1790222782862576 us; error 42 us; skew 500 ppm
Sep 24 04:06:22 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:22.865686 16321 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:06:22.866786 16321 webserver.cc:460] Webserver started at http://127.0.98.62:33357/ using document root <none> and password file <none>
I20260924 04:06:22.867374 16321 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:22.867450 16321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:22.871692 16321 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:06:22.874287 16337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:22.876113 16321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:06:22.876322 16321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "a576b50844ec4ce7a1794a002dc7558e"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "5dcaaeb6fbee1dc7b2e8b7d168421a1c"
server_key_iv: "c34dffeef51bfba71d6cccef7e6d299d"
server_key_version: "encryptionkey@0"
I20260924 04:06:22.877094 16321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:22.892380 16321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:22.895746 16321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:22.896013 16321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:22.903266 16321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38667
I20260924 04:06:22.903565 16389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38667 every 8 connection(s)
I20260924 04:06:22.904484 16321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:06:22.906647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 16321
I20260924 04:06:22.907104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39039
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38667
--webserver_port=39269
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:06:22.909322 16390 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:06:22.911124 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Bootstrap starting.
I20260924 04:06:22.928413 16390 log.cc:824] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:22.938467 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:06:22.938835 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Bootstrap complete.
I20260924 04:06:22.942787 16390 raft_consensus.cc:348] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:22.943188 16390 raft_consensus.cc:729] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a576b50844ec4ce7a1794a002dc7558e, State: Initialized, Role: FOLLOWER
I20260924 04:06:22.943696 16390 consensus_queue.cc:260] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:22.943825 16390 raft_consensus.cc:388] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:06:22.943921 16390 raft_consensus.cc:482] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:06:22.944023 16390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:06:22.945961 16390 raft_consensus.cc:504] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:22.946300 16390 leader_election.cc:302] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a576b50844ec4ce7a1794a002dc7558e; no voters:
I20260924 04:06:22.946591 16390 leader_election.cc:288] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:06:22.946728 16394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:06:22.946995 16394 raft_consensus.cc:686] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 2 LEADER]: Becoming Leader. State: Replica: a576b50844ec4ce7a1794a002dc7558e, State: Running, Role: LEADER
I20260924 04:06:22.947391 16394 consensus_queue.cc:237] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:22.947835 16390 sys_catalog.cc:564] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:06:22.948849 16395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a576b50844ec4ce7a1794a002dc7558e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } } }
I20260924 04:06:22.948972 16395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: This master's current role is: LEADER
I20260924 04:06:22.949838 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a576b50844ec4ce7a1794a002dc7558e. Latest consensus state: current_term: 2 leader_uuid: "a576b50844ec4ce7a1794a002dc7558e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } } }
I20260924 04:06:22.950238 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: This master's current role is: LEADER
I20260924 04:06:22.952838 16410 catalog_manager.cc:1259] Loaded cluster ID: 6abd2432b0344acfb44a82e5fde11a50
I20260924 04:06:22.952903 16410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: loading cluster ID for follower catalog manager: success
I20260924 04:06:22.955466 16410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: acquiring CA information for follower catalog manager: success
I20260924 04:06:22.957100 16410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:06:22.957310 16403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:06:22.959313 16403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:06:22.959748 16403 catalog_manager.cc:1259] Loaded cluster ID: 6abd2432b0344acfb44a82e5fde11a50
I20260924 04:06:22.959805 16403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:06:22.960529 16403 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:06:22.961061 16403 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Loaded TSK: 0
I20260924 04:06:22.974668 16403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Generated new TSK 1
I20260924 04:06:22.974964 16403 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:06:23.026600 16392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:23.026937 16392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:23.027004 16392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:23.031562 16392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:23.031662 16392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:23.031762 16392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:06:23.037467 16392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39269
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:23.038571 16392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:23.039640 16392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:23.049629 16420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:23.049724 16417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:23.049631 16418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:23.049921 16392 server_base.cc:1034] running on GCE node
I20260924 04:06:23.050490 16392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:23.051393 16392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:23.052582 16392 hybrid_clock.cc:648] HybridClock initialized: now 1790222783052545 us; error 51 us; skew 500 ppm
Sep 24 04:06:23 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:23.056196 16392 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:06:23.057439 16392 webserver.cc:460] Webserver started at http://127.0.98.1:39269/ using document root <none> and password file <none>
I20260924 04:06:23.058184 16392 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:23.058303 16392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:23.063488 16392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:06:23.065986 16427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:23.067423 16392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:06:23.067549 16392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "db685c3bb2334df099ed99538d7fd8be"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "22c8c000fccedec28805dd5afb00635b"
server_key_iv: "481c9f5ff1af64ebcdceb72dc7a915a0"
server_key_version: "encryptionkey@0"
I20260924 04:06:23.068195 16392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:23.089684 16392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:23.093008 16392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:23.093202 16392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:23.093843 16392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:23.094964 16434 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:06:23.100093 16392 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:06:23.100173 16392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 04:06:23.100242 16392 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:06:23.102008 16392 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:06:23.102059 16392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:06:23.102378 16434 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Bootstrap starting.
I20260924 04:06:23.111362 16392 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39039
I20260924 04:06:23.111433 16541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39039 every 8 connection(s)
I20260924 04:06:23.112416 16392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:06:23.113729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 16392
I20260924 04:06:23.114182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40053
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38667
--webserver_port=34367
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:06:23.122619 16434 log.cc:824] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:06:23 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:06:23.130852 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.114911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46807 (local address 127.0.98.62:38667)
0924 04:06:23.116142 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.116157 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.116198 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:06:23.117177 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.117188 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.117498 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.117827 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:23.119010 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.120081 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.121277 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.121874 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.122029 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.124847 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:23.124853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:23.124871 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:23.124906 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:23.127057 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.127744 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.127751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.127757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.127945 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.128382 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.128386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.128387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.128831 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:23.128931 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.129645 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.129964 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":114,"threads_started":1}
I20260924 04:06:23.131641 16434 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:06:23.132174 16434 tablet_bootstrap.cc:492] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Bootstrap complete.
I20260924 04:06:23.132418 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.114745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:46807)
0924 04:06:23.115501 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.115560 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.116769 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.117895 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.117904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.118346 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:23.118827 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.118845 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.120238 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.120241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.121150 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.121160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.121995 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.121999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.122190 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.122751 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:23.122773 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:23.124599 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:23.127191 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:23.127198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:23.127214 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:23.127603 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:23.128075 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:23.128078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:23.128079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:23.128253 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:23.129009 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:23.129015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:23.129318 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.131212 (+ 1894us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.131542 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":132,"threads_started":1}
I20260924 04:06:23.133517 16434 ts_tablet_manager.cc:1397] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.014s
I20260924 04:06:23.136373 16542 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38667
I20260924 04:06:23.136672 16542 heartbeater.cc:464] Registering TS with master...
I20260924 04:06:23.136752 16434 raft_consensus.cc:348] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:23.137375 16434 raft_consensus.cc:729] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db685c3bb2334df099ed99538d7fd8be, State: Initialized, Role: FOLLOWER
I20260924 04:06:23.137493 16542 heartbeater.cc:511] Master 127.0.98.62:38667 requested a full tablet report, sending...
I20260924 04:06:23.137961 16434 consensus_queue.cc:260] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } } peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } }
I20260924 04:06:23.139108 16434 ts_tablet_manager.cc:1428] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:06:23.141402 16354 ts_manager.cc:194] Registered new tserver with Master: db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039)
I20260924 04:06:23.145853 16354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46807
Sep 24 04:06:23 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:06:23.153875 16542 heartbeater.cc:503] Master 127.0.98.62:38667 was elected leader, sending a full tablet report...
I20260924 04:06:23.156343 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.145179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39039 (local address 127.0.0.1:44550)
0924 04:06:23.145621 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.145671 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.145817 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.146839 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.146845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.147228 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:23.147385 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.147394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.148787 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.148790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.149536 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.149547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.150104 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.150123 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.150232 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.150731 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:23.150743 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:23.152228 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:23.154398 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:23.154409 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:23.154415 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:23.154712 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:23.155069 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:23.155073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:23.155075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:23.155166 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:23.155693 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:23.155700 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:23.155846 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.156000 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.156206 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":147,"threads_started":1}
I20260924 04:06:23.157085 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.146023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.0.98.1:39039)
0924 04:06:23.146345 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.146356 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.146381 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:06:23.146454 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.146458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.146660 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.147544 (+ 884us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:23.147681 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.148672 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.149645 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.150009 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.150104 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.152388 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:23.152397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:23.152409 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:23.152435 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:23.154275 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.154864 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.154870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.154872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.154964 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.155277 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.155280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.155282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.155569 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:23.156731 (+ 1162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.156827 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.156964 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":126,"threads_started":1}
I20260924 04:06:23.157968 16476 tablet_service.cc:1514] Processing DeleteTablet for tablet e201a98af6764829b8287eb93256d88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:06:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44550
I20260924 04:06:23.158785 16551 tablet_replica.cc:330] stopping tablet replica
I20260924 04:06:23.159068 16551 raft_consensus.cc:2229] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:06:23.159308 16551 raft_consensus.cc:2258] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:06:23.160331 16551 ts_tablet_manager.cc:1905] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:06:23.164096 16551 ts_tablet_manager.cc:1918] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:06:23.164202 16551 log.cc:1196] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal/wals/e201a98af6764829b8287eb93256d88f
I20260924 04:06:23.164510 16551 ts_tablet_manager.cc:1939] T e201a98af6764829b8287eb93256d88f P db685c3bb2334df099ed99538d7fd8be: Deleting consensus metadata
I20260924 04:06:23.166371 16338 catalog_manager.cc:4841] TS db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039): tablet e201a98af6764829b8287eb93256d88f (table test-table [id=bbfbcde17aec47e892266f5fb77998ab]) successfully deleted
W20260924 04:06:23.247030 16546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:23.247323 16546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:23.247380 16546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:23.251672 16546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:23.251756 16546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:23.251820 16546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:06:23.257397 16546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34367
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:23.258484 16546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:23.259534 16546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:23.270152 16556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:23.270139 16559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:23.270210 16546 server_base.cc:1034] running on GCE node
W20260924 04:06:23.270160 16557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:23.270865 16546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:23.271579 16546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:23.272771 16546 hybrid_clock.cc:648] HybridClock initialized: now 1790222783272726 us; error 66 us; skew 500 ppm
Sep 24 04:06:23 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:23.276001 16546 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:06:23.277009 16546 webserver.cc:460] Webserver started at http://127.0.98.2:34367/ using document root <none> and password file <none>
I20260924 04:06:23.277585 16546 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:23.277671 16546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:23.281960 16546 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:06:23.284304 16566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:23.286013 16546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:06:23.286166 16546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "fa80f2c31eea49a9a84b277d29502622"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "dc8d5cd40ca5632e1d437b727d47a41f"
server_key_iv: "b9a88ddb12e655ce92fdc512724f82d9"
server_key_version: "encryptionkey@0"
I20260924 04:06:23.286831 16546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:23.302291 16546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:23.305593 16546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:23.305807 16546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:23.306455 16546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:23.307340 16546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:06:23.307407 16546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:23.307467 16546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:06:23.307497 16546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:23.316643 16546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40053
I20260924 04:06:23.316669 16679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40053 every 8 connection(s)
I20260924 04:06:23.317660 16546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:06:23.322711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 16546
I20260924 04:06:23.323177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37963
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38667
--webserver_port=43487
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:06:23 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:06:23.332465 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.319458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55331 (local address 127.0.98.62:38667)
0924 04:06:23.319708 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.319713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.320728 (+ 1015us) server_negotiation.cc:407] Connection header received
0924 04:06:23.321643 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.321646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.321696 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.321771 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:23.323170 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.323821 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.325188 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.325544 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.325672 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.328495 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:23.328500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:23.328503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:23.328544 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:23.330187 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.330711 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.330715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.330717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.330817 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.331165 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.331168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.331170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.331432 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:23.331541 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.332116 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.332268 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:06:23.333371 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.319888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:55331)
0924 04:06:23.320579 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.320621 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.321417 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.321931 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.321938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.322321 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:23.322978 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.323000 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.323978 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.323981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.325055 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.325065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.325713 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.325716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.325883 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.326448 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:23.326465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:23.328290 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:23.330316 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:23.330322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:23.330335 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:23.330597 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:23.330914 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:23.330916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:23.330918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:23.331064 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:23.331534 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:23.331540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:23.331804 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.332012 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.332317 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":108,"threads_started":1}
I20260924 04:06:23.334664 16680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38667
I20260924 04:06:23.334954 16680 heartbeater.cc:464] Registering TS with master...
I20260924 04:06:23.335589 16680 heartbeater.cc:511] Master 127.0.98.62:38667 requested a full tablet report, sending...
I20260924 04:06:23.337152 16354 ts_manager.cc:194] Registered new tserver with Master: fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053)
I20260924 04:06:23.337776 16354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55331
W20260924 04:06:23.441159 16683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:23.441460 16683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:23.441504 16683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:23.445680 16683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:23.445767 16683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:23.445829 16683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:06:23.452271 16683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43487
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:23.453392 16683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:23.454448 16683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:23.464390 16691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:23.464403 16689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:23.464412 16688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:23.464562 16683 server_base.cc:1034] running on GCE node
I20260924 04:06:23.465123 16683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:23.465968 16683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:23.467139 16683 hybrid_clock.cc:648] HybridClock initialized: now 1790222783467117 us; error 41 us; skew 500 ppm
Sep 24 04:06:23 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:23.470086 16683 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:06:23.471000 16683 webserver.cc:460] Webserver started at http://127.0.98.3:43487/ using document root <none> and password file <none>
I20260924 04:06:23.471556 16683 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:23.471645 16683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:23.475893 16683 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:06:23.478168 16698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:23.479553 16683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:06:23.479669 16683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "1ff5879648834db5b6a1a834ead5a00f"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "43945f44901297a764abf5441cd08ecc"
server_key_iv: "64ffa037b37f25aa3724e91e276d16a9"
server_key_version: "encryptionkey@0"
I20260924 04:06:23.480299 16683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:23.501732 16683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:23.505146 16683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:23.505357 16683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:23.505991 16683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:23.506917 16683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:06:23.506970 16683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:23.507025 16683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:06:23.507052 16683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:23.515980 16683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37963
I20260924 04:06:23.516017 16811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37963 every 8 connection(s)
I20260924 04:06:23.517041 16683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:06:23.520183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 16683
Sep 24 04:06:23 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:06:23.528533 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.520847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.0.98.62:38667)
0924 04:06:23.521071 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.521076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.521235 (+ 159us) server_negotiation.cc:407] Connection header received
0924 04:06:23.521380 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.521384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.521425 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.521498 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:23.522223 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.522772 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.523597 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.523818 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.523925 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.525411 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:23.525415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:23.525418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:23.525446 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:23.526793 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.527248 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.527250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.527252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.527336 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.527643 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.527646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.527648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.527840 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:23.528057 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.528217 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.528364 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260924 04:06:23.530845 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.518940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43597 (local address 127.0.98.62:38667)
0924 04:06:23.519073 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.519077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.520233 (+ 1156us) server_negotiation.cc:407] Connection header received
0924 04:06:23.521318 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.521322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.521396 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.521466 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:23.522651 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.523168 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.524413 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.524724 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.524840 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.527332 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:23.527335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:23.527338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:23.527359 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:23.528871 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.529393 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.529396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.529398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.529474 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.529838 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.529841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.529842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.530010 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:23.530084 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.530622 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.530728 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":2432}
I20260924 04:06:23.531744 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.519266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:43597)
0924 04:06:23.520067 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.520114 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.521083 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.521673 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.521680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.522036 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:23.522511 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.522524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.523280 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.523283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.524264 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.524274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.524893 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.524896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.525059 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.525683 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:23.525700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:23.527175 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:23.528989 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:23.528995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:23.529013 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:23.529299 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:23.529554 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:23.529557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:23.529559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:23.529744 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:23.530109 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:23.530115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:23.530409 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.530573 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.530843 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":111,"threads_started":1}
I20260924 04:06:23.532860 16812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38667
I20260924 04:06:23.533077 16812 heartbeater.cc:464] Registering TS with master...
I20260924 04:06:23.533540 16812 heartbeater.cc:511] Master 127.0.98.62:38667 requested a full tablet report, sending...
I20260924 04:06:23.534493 16354 ts_manager.cc:194] Registered new tserver with Master: 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
I20260924 04:06:23.535076 16354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43597
I20260924 04:06:23.542084 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:06:23.546458 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.542686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.0.98.62:38667)
0924 04:06:23.542826 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.542831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.543033 (+ 202us) server_negotiation.cc:407] Connection header received
0924 04:06:23.543206 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.543209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.543324 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.543417 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:06:23.543676 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.544265 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.545182 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.545415 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.545502 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.545829 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:06:23.546011 (+ 182us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 04:06:23.546306 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35444: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 04:06:23.546551 16545 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35444: token expired
W20260924 04:06:23.546904 16282 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:38667)
I20260924 04:06:23.554283 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.547296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.0.98.62:38667)
0924 04:06:23.547418 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.547421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.547435 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:06:23.547552 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.547555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.547600 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.547663 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:23.548407 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.548951 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.549739 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.549959 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.550043 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.551312 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:23.551316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:23.551319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:23.551345 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:23.552756 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.553139 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.553142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.553144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.553225 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:23.553473 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:23.553476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:23.553478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:23.553659 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:23.553776 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.554006 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.554120 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:06:23.557050 16354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:06:23.560385 16354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:06:23.568389 16476 tablet_service.cc:1467] Processing CreateTablet for tablet d78a6b9ea68c4a3387efe7d6c778e5fa (DEFAULT_TABLE table=test-table [id=9ff1ffebaf76471099686dd6b6cf77dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:06:23.569068 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78a6b9ea68c4a3387efe7d6c778e5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:06:23.571834 16434 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Bootstrap starting.
I20260924 04:06:23.572784 16434 tablet_bootstrap.cc:654] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Neither blocks nor log segments found. Creating new log.
I20260924 04:06:23.573357 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.567717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37963 (local address 127.0.0.1:34096)
0924 04:06:23.568098 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.568111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.568204 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.568950 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.568953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.568972 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:23.569065 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.569071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.570433 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.570436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.571651 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.571672 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.572845 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.572849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.572948 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.573010 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.573101 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.573196 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":17,"thread_start_us":115,"threads_started":1}
I20260924 04:06:23.573382 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.568165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.0.98.3:37963)
0924 04:06:23.568481 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.568494 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.568518 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:06:23.568592 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.568597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.568806 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.568971 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:23.569221 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.570315 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.571795 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.572687 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.572846 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.573070 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.573130 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.573233 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260924 04:06:23.573566 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.567224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40053 (local address 127.0.0.1:49566)
0924 04:06:23.567427 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.567441 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.567516 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.568706 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.568709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.568728 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:23.568838 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.568845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.570277 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.570280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.571651 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.571663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.573201 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.573205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.573304 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.573327 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.573375 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.573455 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":43}
I20260924 04:06:23.574291 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.567658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.0.98.2:40053)
0924 04:06:23.568014 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.568026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.568051 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:06:23.568140 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.568146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.568516 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.568973 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:23.569142 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.570171 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.571795 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.573064 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.573887 (+ 823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.574033 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.574084 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.574179 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":116,"threads_started":1}
I20260924 04:06:23.574857 16434 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: No bootstrap required, opened a new log
I20260924 04:06:23.574954 16434 ts_tablet_manager.cc:1397] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:06:23.575428 16434 raft_consensus.cc:348] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.575551 16434 raft_consensus.cc:374] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:06:23.575593 16434 raft_consensus.cc:729] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db685c3bb2334df099ed99538d7fd8be, State: Initialized, Role: FOLLOWER
I20260924 04:06:23.575773 16434 consensus_queue.cc:260] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.576535 16434 ts_tablet_manager.cc:1428] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:06:23.576926 16746 tablet_service.cc:1467] Processing CreateTablet for tablet d78a6b9ea68c4a3387efe7d6c778e5fa (DEFAULT_TABLE table=test-table [id=9ff1ffebaf76471099686dd6b6cf77dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:06:23.577023 16614 tablet_service.cc:1467] Processing CreateTablet for tablet d78a6b9ea68c4a3387efe7d6c778e5fa (DEFAULT_TABLE table=test-table [id=9ff1ffebaf76471099686dd6b6cf77dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:06:23.578101 16746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78a6b9ea68c4a3387efe7d6c778e5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:06:23.578097 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78a6b9ea68c4a3387efe7d6c778e5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:06:23.586611 16825 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Bootstrap starting.
I20260924 04:06:23.586606 16826 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Bootstrap starting.
I20260924 04:06:23.589267 16826 tablet_bootstrap.cc:654] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Neither blocks nor log segments found. Creating new log.
I20260924 04:06:23.589721 16825 tablet_bootstrap.cc:654] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Neither blocks nor log segments found. Creating new log.
I20260924 04:06:23.590675 16825 log.cc:824] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:23.590675 16826 log.cc:824] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:23.593106 16825 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: No bootstrap required, opened a new log
I20260924 04:06:23.593104 16826 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: No bootstrap required, opened a new log
I20260924 04:06:23.593333 16826 ts_tablet_manager.cc:1397] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 04:06:23.593333 16825 ts_tablet_manager.cc:1397] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:06:23.597404 16825 raft_consensus.cc:348] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.597404 16826 raft_consensus.cc:348] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.597736 16825 raft_consensus.cc:374] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:06:23.597736 16826 raft_consensus.cc:374] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:06:23.597797 16826 raft_consensus.cc:729] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff5879648834db5b6a1a834ead5a00f, State: Initialized, Role: FOLLOWER
I20260924 04:06:23.597805 16825 raft_consensus.cc:729] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa80f2c31eea49a9a84b277d29502622, State: Initialized, Role: FOLLOWER
I20260924 04:06:23.598325 16825 consensus_queue.cc:260] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.598325 16826 consensus_queue.cc:260] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.599124 16812 heartbeater.cc:503] Master 127.0.98.62:38667 was elected leader, sending a full tablet report...
I20260924 04:06:23.599398 16680 heartbeater.cc:503] Master 127.0.98.62:38667 was elected leader, sending a full tablet report...
I20260924 04:06:23.599469 16826 ts_tablet_manager.cc:1428] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:06:23.599469 16825 ts_tablet_manager.cc:1428] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260924 04:06:23.613607 16543 tablet.cc:2369] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:06:23.768122 16813 tablet.cc:2369] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:06:23.818810 16681 tablet.cc:2369] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:06:23.889704 16830 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:06:23.889941 16830 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.890216 16829 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:06:23.890388 16829 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.891455 16830 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039)
I20260924 04:06:23.891476 16829 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963), db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039)
I20260924 04:06:23.896021 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.891426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40053 (local address 127.0.98.3:42449)
0924 04:06:23.891620 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.891635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.891731 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.892236 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.892239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.892258 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:23.892373 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.892380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.893311 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.893314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.894544 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.894554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.895520 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.895523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.895673 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.895697 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.895760 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.895836 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 04:06:23.896445 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.891426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39039 (local address 127.0.98.3:41035)
0924 04:06:23.892089 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.892103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.892205 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.892624 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.892627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.892644 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:23.892756 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.892762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.893756 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.893759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.894942 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.894952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.895978 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.895982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.896155 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.896174 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.896229 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.896295 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":173,"thread_start_us":96,"threads_started":1}
I20260924 04:06:23.896581 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.892338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57727 (local address 127.0.98.3:37963)
0924 04:06:23.892571 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.892575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.892587 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:06:23.892670 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.892673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.892707 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.892749 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:23.893081 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.893794 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.895296 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.895868 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.895966 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.896055 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.896391 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.896454 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":170}
I20260924 04:06:23.896718 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.891796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42449 (local address 127.0.98.2:40053)
0924 04:06:23.891924 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.891929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.891942 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:06:23.891990 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.891993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.892051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.892119 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:23.892499 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.893206 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.894666 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.895317 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.895405 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.895465 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.896515 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.896590 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":33}
I20260924 04:06:23.896930 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.891562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41035 (local address 127.0.98.1:39039)
0924 04:06:23.891828 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.891832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.892287 (+ 455us) server_negotiation.cc:407] Connection header received
0924 04:06:23.892347 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.892350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.892411 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.892509 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:23.892867 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.893642 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.895083 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.895741 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.895837 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.896698 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.896738 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.896804 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260924 04:06:23.896942 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.892051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37963 (local address 127.0.98.2:57727)
0924 04:06:23.892478 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.892490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.892571 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.892852 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.892855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.892870 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:23.892971 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.892977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.893914 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.893917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.895176 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.895185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.896170 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.896173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.896266 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.896287 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.896480 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.896805 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":7,"thread_start_us":72,"threads_started":1}
I20260924 04:06:23.897290 16634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78a6b9ea68c4a3387efe7d6c778e5fa" candidate_uuid: "1ff5879648834db5b6a1a834ead5a00f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa80f2c31eea49a9a84b277d29502622" is_pre_election: true
I20260924 04:06:23.897447 16766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78a6b9ea68c4a3387efe7d6c778e5fa" candidate_uuid: "fa80f2c31eea49a9a84b277d29502622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff5879648834db5b6a1a834ead5a00f" is_pre_election: true
I20260924 04:06:23.897540 16634 raft_consensus.cc:2454] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ff5879648834db5b6a1a834ead5a00f in term 0.
I20260924 04:06:23.897655 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.892370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58245 (local address 127.0.98.1:39039)
0924 04:06:23.892741 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.892745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.892954 (+ 209us) server_negotiation.cc:407] Connection header received
0924 04:06:23.893106 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.893109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.893145 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.893195 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:06:23.893548 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.894284 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.895686 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.896281 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.896364 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.896529 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.897440 (+ 911us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.897518 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":54,"threads_started":1}
I20260924 04:06:23.897680 16766 raft_consensus.cc:2454] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa80f2c31eea49a9a84b277d29502622 in term 0.
I20260924 04:06:23.898002 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.892063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39039 (local address 127.0.98.2:58245)
0924 04:06:23.892878 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.892890 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:23.893011 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:23.893308 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:23.893311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:23.893325 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:06:23.893420 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.893426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.894412 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.894414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.895560 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:23.895568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.897220 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.897223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:23.897313 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.897333 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:06:23.897505 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 04:06:23.897674 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":72,"threads_started":1}
I20260924 04:06:23.897977 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78a6b9ea68c4a3387efe7d6c778e5fa" candidate_uuid: "1ff5879648834db5b6a1a834ead5a00f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db685c3bb2334df099ed99538d7fd8be" is_pre_election: true
I20260924 04:06:23.898108 16569 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff5879648834db5b6a1a834ead5a00f, fa80f2c31eea49a9a84b277d29502622; no voters:
I20260924 04:06:23.898293 16496 raft_consensus.cc:2454] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ff5879648834db5b6a1a834ead5a00f in term 0.
I20260924 04:06:23.898414 16829 raft_consensus.cc:2781] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:06:23.898440 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78a6b9ea68c4a3387efe7d6c778e5fa" candidate_uuid: "fa80f2c31eea49a9a84b277d29502622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db685c3bb2334df099ed99538d7fd8be" is_pre_election: true
I20260924 04:06:23.898494 16829 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:06:23.898547 16829 raft_consensus.cc:3037] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:06:23.898547 16495 raft_consensus.cc:2454] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa80f2c31eea49a9a84b277d29502622 in term 0.
I20260924 04:06:23.898645 16701 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff5879648834db5b6a1a834ead5a00f, fa80f2c31eea49a9a84b277d29502622; no voters:
I20260924 04:06:23.898953 16830 raft_consensus.cc:2781] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:06:23.899027 16830 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:06:23.899068 16830 raft_consensus.cc:3037] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:06:23.899708 16829 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.900084 16829 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [CANDIDATE]: Term 1 election: Requested vote from peers 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963), db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039)
I20260924 04:06:23.900061 16830 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.900391 16830 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [CANDIDATE]: Term 1 election: Requested vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039)
I20260924 04:06:23.900436 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78a6b9ea68c4a3387efe7d6c778e5fa" candidate_uuid: "fa80f2c31eea49a9a84b277d29502622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db685c3bb2334df099ed99538d7fd8be"
I20260924 04:06:23.900488 16766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78a6b9ea68c4a3387efe7d6c778e5fa" candidate_uuid: "fa80f2c31eea49a9a84b277d29502622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff5879648834db5b6a1a834ead5a00f"
I20260924 04:06:23.900547 16495 raft_consensus.cc:3037] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:06:23.900652 16766 raft_consensus.cc:2379] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fa80f2c31eea49a9a84b277d29502622 in current term 1: Already voted for candidate 1ff5879648834db5b6a1a834ead5a00f in this term.
I20260924 04:06:23.900679 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78a6b9ea68c4a3387efe7d6c778e5fa" candidate_uuid: "1ff5879648834db5b6a1a834ead5a00f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db685c3bb2334df099ed99538d7fd8be"
I20260924 04:06:23.900771 16634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78a6b9ea68c4a3387efe7d6c778e5fa" candidate_uuid: "1ff5879648834db5b6a1a834ead5a00f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa80f2c31eea49a9a84b277d29502622"
I20260924 04:06:23.900926 16634 raft_consensus.cc:2379] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ff5879648834db5b6a1a834ead5a00f in current term 1: Already voted for candidate fa80f2c31eea49a9a84b277d29502622 in this term.
I20260924 04:06:23.901273 16701 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ff5879648834db5b6a1a834ead5a00f; no voters: db685c3bb2334df099ed99538d7fd8be, fa80f2c31eea49a9a84b277d29502622
I20260924 04:06:23.901434 16830 raft_consensus.cc:2726] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 04:06:23.901731 16495 raft_consensus.cc:2454] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa80f2c31eea49a9a84b277d29502622 in term 1.
I20260924 04:06:23.901950 16567 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db685c3bb2334df099ed99538d7fd8be, fa80f2c31eea49a9a84b277d29502622; no voters: 1ff5879648834db5b6a1a834ead5a00f
I20260924 04:06:23.902107 16829 raft_consensus.cc:2781] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:06:23.902307 16829 raft_consensus.cc:686] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 1 LEADER]: Becoming Leader. State: Replica: fa80f2c31eea49a9a84b277d29502622, State: Running, Role: LEADER
I20260924 04:06:23.902649 16829 consensus_queue.cc:237] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:23.905575 16354 catalog_manager.cc:5495] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa80f2c31eea49a9a84b277d29502622 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fa80f2c31eea49a9a84b277d29502622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } health_report { overall_health: UNKNOWN } } }
I20260924 04:06:23.938457 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.934857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.0.98.2:40053)
0924 04:06:23.934979 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:23.934982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:23.934995 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:06:23.935074 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:23.935076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:23.935131 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:23.935250 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:06:23.935524 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.936063 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.936904 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:23.937126 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:23.937214 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:23.937527 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:06:23.938042 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:06:23.938161 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:23.938248 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 04:06:23.938306 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:06:23.945664 16766 raft_consensus.cc:1261] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Refusing update from remote peer fa80f2c31eea49a9a84b277d29502622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:06:23.945665 16495 raft_consensus.cc:1261] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Refusing update from remote peer fa80f2c31eea49a9a84b277d29502622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:06:23.946586 16836 consensus_queue.cc:1035] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:06:23.946838 16829 consensus_queue.cc:1035] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:06:23.956173 16839 mvcc.cc:205] Tried to move back new op lower bound from 7332752523032952832 to 7332752522867752960. Current Snapshot: MvccSnapshot[applied={T|T < 7332752523032952832 or (T in {7332752523032952832})}]
I20260924 04:06:23.958348 16841 mvcc.cc:205] Tried to move back new op lower bound from 7332752523032952832 to 7332752522867752960. Current Snapshot: MvccSnapshot[applied={T|T < 7332752523032952832 or (T in {7332752523032952832})}]
I20260924 04:06:23.964754 16354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:06:23.965001 16354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:06:23.967978 16354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Sending DeleteTablet for 3 replicas of tablet d78a6b9ea68c4a3387efe7d6c778e5fa
I20260924 04:06:23.968902 16476 tablet_service.cc:1514] Processing DeleteTablet for tablet d78a6b9ea68c4a3387efe7d6c778e5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:06:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44550
I20260924 04:06:23.968945 16614 tablet_service.cc:1514] Processing DeleteTablet for tablet d78a6b9ea68c4a3387efe7d6c778e5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:06:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49566
I20260924 04:06:23.968982 16746 tablet_service.cc:1514] Processing DeleteTablet for tablet d78a6b9ea68c4a3387efe7d6c778e5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:06:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34096
I20260924 04:06:23.969789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 16392
I20260924 04:06:23.969823 16851 tablet_replica.cc:330] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: stopping tablet replica
I20260924 04:06:23.971680 16851 raft_consensus.cc:2229] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:06:23.972114 16851 raft_consensus.cc:2258] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:06:23.972675 16849 tablet_replica.cc:330] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: stopping tablet replica
I20260924 04:06:23.973066 16849 raft_consensus.cc:2229] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:06:23.973526 16849 raft_consensus.cc:2258] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:06:23.974745 16851 ts_tablet_manager.cc:1905] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:06:23.975111 16849 ts_tablet_manager.cc:1905] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:06:23.977878 16338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39039 (error 108)
I20260924 04:06:23.978271 16849 ts_tablet_manager.cc:1918] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:06:23.978410 16849 log.cc:1196] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/d78a6b9ea68c4a3387efe7d6c778e5fa
I20260924 04:06:23.978758 16849 ts_tablet_manager.cc:1939] T d78a6b9ea68c4a3387efe7d6c778e5fa P fa80f2c31eea49a9a84b277d29502622: Deleting consensus metadata
I20260924 04:06:23.978762 16851 ts_tablet_manager.cc:1918] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:06:23.978991 16851 log.cc:1196] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/d78a6b9ea68c4a3387efe7d6c778e5fa
I20260924 04:06:23.979403 16851 ts_tablet_manager.cc:1939] T d78a6b9ea68c4a3387efe7d6c778e5fa P 1ff5879648834db5b6a1a834ead5a00f: Deleting consensus metadata
I20260924 04:06:23.979717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 16546
I20260924 04:06:23.979846 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.979449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39039 (local address 127.0.0.1:44562)
0924 04:06:23.979663 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.979750 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260924 04:06:23.980398 16338 catalog_manager.cc:4584] TS db685c3bb2334df099ed99538d7fd8be (127.0.98.1:39039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d78a6b9ea68c4a3387efe7d6c778e5fa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39039: connect: Connection refused (error 111)
I20260924 04:06:23.981559 16340 catalog_manager.cc:4841] TS 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963): tablet d78a6b9ea68c4a3387efe7d6c778e5fa (table test-table [id=9ff1ffebaf76471099686dd6b6cf77dd]) successfully deleted
I20260924 04:06:23.985687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 16683
I20260924 04:06:23.986284 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:23.985905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40053 (local address 127.0.0.1:49586)
0924 04:06:23.986136 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:23.986191 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 04:06:23.990895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 16321
I20260924 04:06:43.997236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38667
--webserver_interface=127.0.98.62
--webserver_port=33357
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:06:44.114885 16853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:44.115171 16853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:44.115212 16853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:44.119356 16853 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:06:44.119419 16853 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:06:44.119433 16853 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:06:44.119445 16853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:06:44.119462 16853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:44.119485 16853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:44.119504 16853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:06:44.119537 16853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:06:44.125368 16853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:44.126505 16853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:44.127589 16853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:44.136602 16861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:44.136647 16859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:44.136696 16858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:44.136759 16853 server_base.cc:1034] running on GCE node
I20260924 04:06:44.137367 16853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:44.138530 16853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:44.139709 16853 hybrid_clock.cc:648] HybridClock initialized: now 1790222804139690 us; error 35 us; skew 500 ppm
Sep 24 04:06:44 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:44.142462 16853 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:06:44.143388 16853 webserver.cc:460] Webserver started at http://127.0.98.62:33357/ using document root <none> and password file <none>
I20260924 04:06:44.143996 16853 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:44.144065 16853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:44.147943 16853 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:06:44.150168 16868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:44.151417 16853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:06:44.151541 16853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "a576b50844ec4ce7a1794a002dc7558e"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "5dcaaeb6fbee1dc7b2e8b7d168421a1c"
server_key_iv: "c34dffeef51bfba71d6cccef7e6d299d"
server_key_version: "encryptionkey@0"
I20260924 04:06:44.152137 16853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:44.168893 16853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:44.172292 16853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:44.172457 16853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:44.179716 16853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38667
I20260924 04:06:44.180042 16920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38667 every 8 connection(s)
I20260924 04:06:44.180920 16853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:06:44.184260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 16853
I20260924 04:06:44.184665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39039
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38667
--webserver_port=39269
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:06:44.185608 16921 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:06:44.187516 16921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Bootstrap starting.
I20260924 04:06:44.205235 16921 log.cc:824] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:44.216710 16921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:06:44.217101 16921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Bootstrap complete.
I20260924 04:06:44.221071 16921 raft_consensus.cc:348] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:44.221498 16921 raft_consensus.cc:729] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a576b50844ec4ce7a1794a002dc7558e, State: Initialized, Role: FOLLOWER
I20260924 04:06:44.222021 16921 consensus_queue.cc:260] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:44.222157 16921 raft_consensus.cc:388] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:06:44.222229 16921 raft_consensus.cc:482] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:06:44.222327 16921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:06:44.224349 16921 raft_consensus.cc:504] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:44.224679 16921 leader_election.cc:302] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a576b50844ec4ce7a1794a002dc7558e; no voters:
I20260924 04:06:44.224984 16921 leader_election.cc:288] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:06:44.225096 16925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:06:44.225289 16925 raft_consensus.cc:686] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [term 3 LEADER]: Becoming Leader. State: Replica: a576b50844ec4ce7a1794a002dc7558e, State: Running, Role: LEADER
I20260924 04:06:44.225682 16925 consensus_queue.cc:237] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } }
I20260924 04:06:44.226284 16921 sys_catalog.cc:564] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:06:44.227291 16926 sys_catalog.cc:455] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a576b50844ec4ce7a1794a002dc7558e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } } }
I20260924 04:06:44.227435 16926 sys_catalog.cc:458] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: This master's current role is: LEADER
I20260924 04:06:44.227682 16927 sys_catalog.cc:455] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a576b50844ec4ce7a1794a002dc7558e. Latest consensus state: current_term: 3 leader_uuid: "a576b50844ec4ce7a1794a002dc7558e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a576b50844ec4ce7a1794a002dc7558e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38667 } } }
I20260924 04:06:44.227754 16927 sys_catalog.cc:458] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e [sys.catalog]: This master's current role is: LEADER
I20260924 04:06:44.227830 16934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:06:44.232945 16934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:06:44.233652 16934 catalog_manager.cc:1259] Loaded cluster ID: 6abd2432b0344acfb44a82e5fde11a50
I20260924 04:06:44.233712 16934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:06:44.235494 16934 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:06:44.236253 16934 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Loaded TSK: 0
I20260924 04:06:44.236330 16934 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a576b50844ec4ce7a1794a002dc7558e: Loaded TSK: 1
F20260924 04:06:44.236965 16934 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc9636979ad google::LogMessage::Fail() at ??:0
@ 0x7fc96369a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc9636974de google::LogMessage::Flush() at ??:0
@ 0x7fc963698a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc96635b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc966353f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc96e0007bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc96dfda99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc96dfdb178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc96e01cdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc96e03b82b std::function<>::operator()() at ??:0
@ 0x7fc96dfdab4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc96dfdbf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc96dfd8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc96e01c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc9658e93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc9658e9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc9658eb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc9658da512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc966f1e6db start_thread at ??:0
@ 0x7fc96219561f clone at ??:0
W20260924 04:06:44.330438 16923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:44.330708 16923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:44.330757 16923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:44.334985 16923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:44.335050 16923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:44.335116 16923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:06:44.340677 16923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39269
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:44.341787 16923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:44.342841 16923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:44.352540 16968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:44.352535 16969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:44.352829 16923 server_base.cc:1034] running on GCE node
W20260924 04:06:44.352874 16971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:44.353295 16923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:44.353998 16923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:44.355180 16923 hybrid_clock.cc:648] HybridClock initialized: now 1790222804355148 us; error 47 us; skew 500 ppm
Sep 24 04:06:44 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:44.357930 16923 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:06:44.358925 16923 webserver.cc:460] Webserver started at http://127.0.98.1:39269/ using document root <none> and password file <none>
I20260924 04:06:44.359503 16923 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:44.359566 16923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:44.363448 16923 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:06:44.365597 16978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:44.366887 16923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:06:44.366992 16923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "db685c3bb2334df099ed99538d7fd8be"
format_stamp: "Formatted at 2026-09-24 04:06:01 on dist-test-slave-zcsk"
server_key: "22c8c000fccedec28805dd5afb00635b"
server_key_iv: "481c9f5ff1af64ebcdceb72dc7a915a0"
server_key_version: "encryptionkey@0"
I20260924 04:06:44.367571 16923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:44.378126 16923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:44.381165 16923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:44.381345 16923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:44.381922 16923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:44.383034 16985 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:06:44.387029 16923 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:06:44.387085 16923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 04:06:44.387135 16923 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:06:44.388833 16923 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:06:44.388888 16923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260924 04:06:44.389168 16985 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Bootstrap starting.
I20260924 04:06:44.398939 16923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39039
I20260924 04:06:44.399233 17092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39039 every 8 connection(s)
I20260924 04:06:44.400309 16923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:06:44.401718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 16923
I20260924 04:06:44.402174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40053
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38667
--webserver_port=34367
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:06:44.411528 16985 log.cc:824] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Log is configured to *not* fsync() on all Append() calls
I20260924 04:06:44.421411 16985 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:06:44.421917 16985 tablet_bootstrap.cc:492] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Bootstrap complete.
I20260924 04:06:44.423281 16985 ts_tablet_manager.cc:1397] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.008s
I20260924 04:06:44.426741 16985 raft_consensus.cc:348] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:44.427309 16985 raft_consensus.cc:729] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db685c3bb2334df099ed99538d7fd8be, State: Initialized, Role: FOLLOWER
I20260924 04:06:44.427871 16985 consensus_queue.cc:260] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:44.429030 16985 ts_tablet_manager.cc:1428] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260924 04:06:44.452687 17095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:06:44.402731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:37373)
0924 04:06:44.403969 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.404031 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:44.405055 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:44.451778 (+ 46723us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":141,"threads_started":1}
W20260924 04:06:44.453296 17093 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38667 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38667: Client connection negotiation failed: client connection to 127.0.98.62:38667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:06:44.454161 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:44.453858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:46267)
0924 04:06:44.454021 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.454097 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 04:06:44.454895 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:44.454670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:38791)
0924 04:06:44.454788 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.454839 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:06:44.455057 17093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:06:44.518805 17096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:44.519094 17096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:44.519158 17096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:44.523272 17096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:44.523344 17096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:44.523437 17096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:06:44.528832 17096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34367
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:44.529913 17096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:44.531005 17096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:44.540936 17106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:44.540871 17104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:44.540942 17103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:44.541030 17096 server_base.cc:1034] running on GCE node
I20260924 04:06:44.541502 17096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:44.542167 17096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:44.543375 17096 hybrid_clock.cc:648] HybridClock initialized: now 1790222804543347 us; error 47 us; skew 500 ppm
Sep 24 04:06:44 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:44.546203 17096 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:06:44.547130 17096 webserver.cc:460] Webserver started at http://127.0.98.2:34367/ using document root <none> and password file <none>
I20260924 04:06:44.547691 17096 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:44.547758 17096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:44.551800 17096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:06:44.554070 17113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:44.555495 17096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:06:44.555630 17096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "fa80f2c31eea49a9a84b277d29502622"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "dc8d5cd40ca5632e1d437b727d47a41f"
server_key_iv: "b9a88ddb12e655ce92fdc512724f82d9"
server_key_version: "encryptionkey@0"
I20260924 04:06:44.556348 17096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:44.571004 17096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:44.574484 17096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:44.574687 17096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:44.575313 17096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:44.576239 17096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:06:44.576289 17096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:44.576345 17096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:06:44.576380 17096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:44.585747 17096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40053
I20260924 04:06:44.585777 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40053 every 8 connection(s)
I20260924 04:06:44.586862 17096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:06:44.589753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 17096
I20260924 04:06:44.590188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37963
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38667
--webserver_port=43487
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:06:44.590996 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:44.589276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:34913)
0924 04:06:44.590064 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.590212 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":139,"threads_started":1}
W20260924 04:06:44.591670 17227 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38667 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38667: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
I20260924 04:06:44.592520 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:44.592293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:46153)
0924 04:06:44.592411 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.592460 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:06:44.593178 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:44.592983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:49375)
0924 04:06:44.593081 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.593126 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:06:44.593341 17227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:06:44.703080 17230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:06:44.703390 17230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:06:44.703457 17230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:06:44.707710 17230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:06:44.707796 17230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:06:44.707887 17230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:06:44.713558 17230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43487
--tserver_master_addrs=127.0.98.62:38667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:06:44.714747 17230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:06:44.715832 17230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:06:44.725765 17235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:44.725770 17236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:06:44.725770 17238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:06:44.725832 17230 server_base.cc:1034] running on GCE node
I20260924 04:06:44.726477 17230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:06:44.727111 17230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:06:44.728282 17230 hybrid_clock.cc:648] HybridClock initialized: now 1790222804728263 us; error 35 us; skew 500 ppm
Sep 24 04:06:44 dist-test-slave-zcsk krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:06:44.731127 17230 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:06:44.732100 17230 webserver.cc:460] Webserver started at http://127.0.98.3:43487/ using document root <none> and password file <none>
I20260924 04:06:44.732648 17230 fs_manager.cc:359] Metadata directory not provided
I20260924 04:06:44.732714 17230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:06:44.736755 17230 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 04:06:44.738986 17245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:44.740408 17230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:06:44.740520 17230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "1ff5879648834db5b6a1a834ead5a00f"
format_stamp: "Formatted at 2026-09-24 04:06:02 on dist-test-slave-zcsk"
server_key: "43945f44901297a764abf5441cd08ecc"
server_key_iv: "64ffa037b37f25aa3724e91e276d16a9"
server_key_version: "encryptionkey@0"
I20260924 04:06:44.741117 17230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:06:44.755334 17230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:06:44.758587 17230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:06:44.758766 17230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:06:44.759399 17230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:06:44.760335 17230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:06:44.760383 17230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:44.760437 17230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:06:44.760464 17230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:06:44.769754 17230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37963
I20260924 04:06:44.769815 17358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37963 every 8 connection(s)
I20260924 04:06:44.770788 17230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:06:44.774712 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:44.773155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:35105)
0924 04:06:44.773848 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.773964 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":124,"threads_started":1}
W20260924 04:06:44.775265 17359 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38667 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38667: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
I20260924 04:06:44.775892 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:44.775678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:33745)
0924 04:06:44.775789 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.775837 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:06:44.776513 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:44.776321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:52667)
0924 04:06:44.776416 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:44.776462 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:06:44.776678 17359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:06:44.777402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 17230
I20260924 04:06:45.456158 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:45.455697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:45877)
0924 04:06:45.455978 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:45.456045 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260924 04:06:45.594249 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:45.593861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:39007)
0924 04:06:45.594088 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:45.594141 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260924 04:06:45.777745 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:45.777276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:48409)
0924 04:06:45.777551 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:45.777614 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
I20260924 04:06:46.120939 17366 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:06:46.121133 17366 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:46.122182 17366 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
Sep 24 04:06:46 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 04:06:46 dist-test-slave-zcsk krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 04:06:46.139810 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:46.122511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40053 (local address 127.0.98.1:60273)
0924 04:06:46.123382 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:46.123396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:46.123517 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:46.125432 (+ 1915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:46.125439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:46.126036 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:46.126811 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:46.126833 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:46.128755 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:46.128759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:46.130310 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:46.130317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:46.131315 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:46.131319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:46.131615 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:46.132419 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:46.132449 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:46.134673 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:46.137298 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:46.137304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:46.137319 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:46.137606 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:46.137987 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:46.137989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:46.137991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:46.138133 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:46.138804 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:46.138814 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:46.139068 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 04:06:46.139241 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 04:06:46.139607 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":282,"spinlock_wait_cycles":45952,"thread_start_us":78,"threads_started":1}
I20260924 04:06:46.139839 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:46.122609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60273 (local address 127.0.98.2:40053)
0924 04:06:46.123084 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:46.123102 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:46.123495 (+ 393us) server_negotiation.cc:407] Connection header received
0924 04:06:46.124500 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:46.124523 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:46.124978 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:46.125309 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:46.126999 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:46.128608 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:46.130427 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:46.131170 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:46.131335 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:46.134941 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:46.134948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:46.134962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:46.135012 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:46.137167 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:46.137727 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:46.137735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:46.137742 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:46.137881 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:46.138222 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:46.138226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:46.138228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:46.138682 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:46.138803 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:46.139411 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 04:06:46.139681 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260924 04:06:46.140062 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:46.123054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58677 (local address 127.0.98.3:37963)
0924 04:06:46.123321 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 04:06:46.123346 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 04:06:46.123493 (+ 147us) server_negotiation.cc:407] Connection header received
0924 04:06:46.124500 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:06:46.124508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:06:46.124977 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:06:46.125309 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:06:46.126999 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:46.128607 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:46.130427 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:06:46.131170 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:06:46.131336 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:46.135208 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:06:46.135222 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:06:46.135240 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:06:46.135282 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:06:46.137167 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:46.137757 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:46.137762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:46.137766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:46.137892 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:06:46.138241 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:06:46.138243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:06:46.138245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:06:46.138673 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:06:46.138785 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:06:46.139573 (+ 788us) server_negotiation.cc:299] Negotiation successful
0924 04:06:46.139854 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260924 04:06:46.139985 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:46.122900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37963 (local address 127.0.98.1:58677)
0924 04:06:46.123402 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:46.123413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:06:46.123573 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:06:46.125432 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:06:46.125439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:06:46.126036 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:06:46.126811 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:46.126833 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:46.128755 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:46.128759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:46.130287 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:06:46.130300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:06:46.131315 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:06:46.131319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:06:46.131616 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:06:46.132419 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:06:46.132449 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:06:46.134929 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:06:46.137344 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:46.137348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:46.137351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:46.137629 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:46.138001 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:06:46.138003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:06:46.138004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:06:46.138145 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:06:46.138804 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:06:46.138811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:06:46.139070 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 04:06:46.139243 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 04:06:46.139723 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":441,"spinlock_wait_cycles":50176,"thread_start_us":59,"threads_started":1}
W20260924 04:06:46.142773 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
W20260924 04:06:46.142988 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
I20260924 04:06:46.143056 16981 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db685c3bb2334df099ed99538d7fd8be; no voters: 1ff5879648834db5b6a1a834ead5a00f, fa80f2c31eea49a9a84b277d29502622
I20260924 04:06:46.143360 17366 raft_consensus.cc:2726] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:06:46.457309 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:46.456920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:51383)
0924 04:06:46.457084 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:46.457150 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 04:06:46.595587 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:46.595096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:36715)
0924 04:06:46.595406 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:46.595470 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":74,"threads_started":1}
I20260924 04:06:46.779058 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:46.778564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:45265)
0924 04:06:46.778892 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:46.778957 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20260924 04:06:47.458690 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:47.458207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:45639)
0924 04:06:47.458510 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:47.458587 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20260924 04:06:47.596922 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:47.596465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:48543)
0924 04:06:47.596744 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:47.596820 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260924 04:06:47.780452 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:47.780020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:55645)
0924 04:06:47.780292 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:47.780358 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260924 04:06:47.885447 17376 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:06:47.885584 17376 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:47.885949 17376 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
W20260924 04:06:47.886832 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
W20260924 04:06:47.886967 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
I20260924 04:06:47.887020 16981 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db685c3bb2334df099ed99538d7fd8be; no voters: 1ff5879648834db5b6a1a834ead5a00f, fa80f2c31eea49a9a84b277d29502622
I20260924 04:06:47.887190 17376 raft_consensus.cc:2726] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:06:48.460158 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:48.459666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:40517)
0924 04:06:48.459988 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:48.460056 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260924 04:06:48.598260 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:48.597839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:40161)
0924 04:06:48.598109 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:48.598164 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260924 04:06:48.781857 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:48.781344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:57953)
0924 04:06:48.781682 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:48.781758 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":95,"threads_started":1}
I20260924 04:06:49.461576 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:49.461053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:39019)
0924 04:06:49.461412 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:49.461477 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":73,"threads_started":1}
I20260924 04:06:49.599622 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:49.599122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:37277)
0924 04:06:49.599444 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:49.599518 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260924 04:06:49.684695 17382 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:06:49.684839 17382 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:49.685271 17382 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
W20260924 04:06:49.686216 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
W20260924 04:06:49.686371 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
I20260924 04:06:49.686432 16981 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db685c3bb2334df099ed99538d7fd8be; no voters: 1ff5879648834db5b6a1a834ead5a00f, fa80f2c31eea49a9a84b277d29502622
I20260924 04:06:49.686568 17382 raft_consensus.cc:2726] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:06:49.783128 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:49.782687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:48441)
0924 04:06:49.782967 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:49.783031 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260924 04:06:50.462909 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:50.462429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:45763)
0924 04:06:50.462732 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:50.462795 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":70,"threads_started":1}
I20260924 04:06:50.600957 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:50.600440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:43571)
0924 04:06:50.600759 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:50.600838 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":74,"threads_started":1}
I20260924 04:06:50.784358 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:50.783922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:40773)
0924 04:06:50.784191 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:50.784252 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260924 04:06:51.464219 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:51.463711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:60165)
0924 04:06:51.464025 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:51.464115 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260924 04:06:51.602245 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:51.601732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:41239)
0924 04:06:51.602081 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:51.602146 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":88,"threads_started":1}
I20260924 04:06:51.785501 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:51.785021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:50603)
0924 04:06:51.785345 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:51.785406 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":71,"threads_started":1}
I20260924 04:06:52.465580 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:52.465087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:56181)
0924 04:06:52.465388 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:52.465459 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260924 04:06:52.603544 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:52.603078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:50333)
0924 04:06:52.603376 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:52.603442 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260924 04:06:52.786923 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:52.786359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:40601)
0924 04:06:52.786765 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:52.786830 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":82,"threads_started":1}
I20260924 04:06:53.466913 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:53.466444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:52677)
0924 04:06:53.466752 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:53.466819 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":80,"threads_started":1}
I20260924 04:06:53.604805 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:53.604367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:54135)
0924 04:06:53.604635 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:53.604706 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260924 04:06:53.788347 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:53.787766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:58081)
0924 04:06:53.788142 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:53.788242 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
I20260924 04:06:53.812844 17396 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:06:53.813021 17396 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:53.813378 17396 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
W20260924 04:06:53.814218 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
W20260924 04:06:53.814344 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
I20260924 04:06:53.814393 16981 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db685c3bb2334df099ed99538d7fd8be; no voters: 1ff5879648834db5b6a1a834ead5a00f, fa80f2c31eea49a9a84b277d29502622
I20260924 04:06:53.814571 17396 raft_consensus.cc:2726] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:06:54.468480 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:54.467996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:54425)
0924 04:06:54.468303 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:54.468375 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
I20260924 04:06:54.606019 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:54.605602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:52591)
0924 04:06:54.605867 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:54.605922 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":71,"threads_started":1}
I20260924 04:06:54.789615 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:54.789126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:40563)
0924 04:06:54.789429 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:54.789497 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260924 04:06:55.469805 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:55.469316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:38751)
0924 04:06:55.469627 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:55.469697 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":77,"threads_started":1}
I20260924 04:06:55.607084 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:55.606690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:35975)
0924 04:06:55.606933 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:55.606990 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260924 04:06:55.790861 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:55.790435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:57349)
0924 04:06:55.790690 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:55.790746 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260924 04:06:56.471192 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:56.470658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:36347)
0924 04:06:56.470956 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:56.471041 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260924 04:06:56.608294 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:56.607855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:54013)
0924 04:06:56.608132 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:56.608195 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
I20260924 04:06:56.792186 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:56.791705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:45555)
0924 04:06:56.792022 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:56.792085 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
I20260924 04:06:57.472357 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:57.471962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:43775)
0924 04:06:57.472210 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:57.472261 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260924 04:06:57.609639 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:57.609164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:58907)
0924 04:06:57.609470 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:57.609538 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260924 04:06:57.793354 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:57.792920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:60583)
0924 04:06:57.793200 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:57.793259 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260924 04:06:58.473506 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:58.473082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:59859)
0924 04:06:58.473353 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:58.473413 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260924 04:06:58.610996 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:58.610529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:60783)
0924 04:06:58.610818 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:58.610890 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260924 04:06:58.794564 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:58.794091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.3:34813)
0924 04:06:58.794392 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:58.794458 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260924 04:06:59.163179 17412 raft_consensus.cc:482] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:06:59.163321 17412 raft_consensus.cc:504] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa80f2c31eea49a9a84b277d29502622" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40053 } } peers { permanent_uuid: "1ff5879648834db5b6a1a834ead5a00f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37963 } } peers { permanent_uuid: "db685c3bb2334df099ed99538d7fd8be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39039 } }
I20260924 04:06:59.163786 17412 leader_election.cc:288] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053), 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963)
W20260924 04:06:59.164564 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa80f2c31eea49a9a84b277d29502622 (127.0.98.2:40053): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
W20260924 04:06:59.164729 16981 leader_election.cc:341] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ff5879648834db5b6a1a834ead5a00f (127.0.98.3:37963): Not found: Tablet not found: d78a6b9ea68c4a3387efe7d6c778e5fa
I20260924 04:06:59.164785 16981 leader_election.cc:302] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db685c3bb2334df099ed99538d7fd8be; no voters: 1ff5879648834db5b6a1a834ead5a00f, fa80f2c31eea49a9a84b277d29502622
I20260924 04:06:59.164942 17412 raft_consensus.cc:2726] T d78a6b9ea68c4a3387efe7d6c778e5fa P db685c3bb2334df099ed99538d7fd8be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:06:59.475133 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:59.474560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.1:48501)
0924 04:06:59.474943 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:59.475017 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":91,"threads_started":1}
I20260924 04:06:59.612437 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:06:59.612023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38667 (local address 127.0.98.2:40511)
0924 04:06:59.612273 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:06:59.612337 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260924 04:06:59.782812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 16923
I20260924 04:06:59.787981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 17096
I20260924 04:06:59.792878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 17230
2026-09-24T04:06:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59324 ms)
[----------] 30 tests from SecurityITest (242997 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:06:59 dist-test-slave-zcsk krb5kdc[17417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:06:59 dist-test-slave-zcsk krb5kdc[17417](info): set up 2 sockets
Sep 24 04:06:59 dist-test-slave-zcsk krb5kdc[17417](info): commencing operation
krb5kdc: starting...
W20260924 04:07:00.776059 392 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:00 dist-test-slave-zcsk krb5kdc[17417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:00.894246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35731
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:01.004914 17430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:01.005211 17430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:01.005252 17430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:01.009382 17430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:01.009447 17430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:01.009465 17430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:01.009483 17430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:01.009513 17430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:01.015316 17430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:01.016517 17430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:01.017603 17430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:01.026233 17438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:01.026283 17436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:01.026233 17435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:01.026496 17430 server_base.cc:1034] running on GCE node
I20260924 04:07:01.027096 17430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:01.028275 17430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:01.029444 17430 hybrid_clock.cc:648] HybridClock initialized: now 1790222821029417 us; error 38 us; skew 500 ppm
Sep 24 04:07:01 dist-test-slave-zcsk krb5kdc[17417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:01.032163 17430 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:01.033037 17430 webserver.cc:460] Webserver started at http://127.0.98.62:41055/ using document root <none> and password file <none>
I20260924 04:07:01.033536 17430 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:01.033599 17430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:01.033767 17430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:01.035707 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "9c10660f8f90427283ff9aebdf8644ac"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "c31a8c7f586ce4f5f68eaccde3491e0a"
server_key_iv: "d2da48ce0ae00cc40af84b9478c17ce0"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.036233 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c10660f8f90427283ff9aebdf8644ac"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "c31a8c7f586ce4f5f68eaccde3491e0a"
server_key_iv: "d2da48ce0ae00cc40af84b9478c17ce0"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.039835 17430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:01.042341 17445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.043598 17430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:07:01.043743 17430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "9c10660f8f90427283ff9aebdf8644ac"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "c31a8c7f586ce4f5f68eaccde3491e0a"
server_key_iv: "d2da48ce0ae00cc40af84b9478c17ce0"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.043848 17430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:01.070894 17430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:01.073961 17430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:01.074146 17430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:01.081135 17430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35731
I20260924 04:07:01.081184 17497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35731 every 8 connection(s)
I20260924 04:07:01.082180 17430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:01.085232 17498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:01.090754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 17430
I20260924 04:07:01.090883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:01.091079 17498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac: Bootstrap starting.
I20260924 04:07:01.091166 392 external_mini_cluster.cc:1428] Setting key e930a6557246cedfdca486e7c9633420
I20260924 04:07:01.093675 17498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:01.094458 17498 log.cc:824] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:01.096755 17498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac: No bootstrap required, opened a new log
Sep 24 04:07:01 dist-test-slave-zcsk krb5kdc[17417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:01.101308 17498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10660f8f90427283ff9aebdf8644ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35731 } }
I20260924 04:07:01.101627 17498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:01.101693 17498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c10660f8f90427283ff9aebdf8644ac, State: Initialized, Role: FOLLOWER
I20260924 04:07:01.102203 17498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10660f8f90427283ff9aebdf8644ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35731 } }
I20260924 04:07:01.102360 17498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:01.102427 17498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:01.102524 17498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:01.103601 17498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10660f8f90427283ff9aebdf8644ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35731 } }
I20260924 04:07:01.104007 17498 leader_election.cc:302] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c10660f8f90427283ff9aebdf8644ac; no voters:
I20260924 04:07:01.104334 17498 leader_election.cc:288] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:01.104430 17503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:01.104600 17503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [term 1 LEADER]: Becoming Leader. State: Replica: 9c10660f8f90427283ff9aebdf8644ac, State: Running, Role: LEADER
I20260924 04:07:01.104967 17503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10660f8f90427283ff9aebdf8644ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35731 } }
I20260924 04:07:01.105690 17498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:01.106226 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c10660f8f90427283ff9aebdf8644ac. Latest consensus state: current_term: 1 leader_uuid: "9c10660f8f90427283ff9aebdf8644ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10660f8f90427283ff9aebdf8644ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35731 } } }
I20260924 04:07:01.106336 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:01.106482 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.092558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.0.98.62:35731)
0924 04:07:01.092954 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.092970 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.093006 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:07:01.093688 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.093707 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.094112 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.094410 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:01.095285 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.096347 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.097490 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.097982 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.098136 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.101048 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:01.101053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:01.101070 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:01.101101 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:01.102930 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.103408 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.103411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.103416 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.103523 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.103825 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.103828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.103831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.104377 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:01.105176 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.105308 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.105541 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260924 04:07:01.106886 17504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c10660f8f90427283ff9aebdf8644ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10660f8f90427283ff9aebdf8644ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35731 } } }
I20260924 04:07:01.107038 17504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac [sys.catalog]: This master's current role is: LEADER
W20260924 04:07:01.110723 17518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:01.110797 17518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:01.110867 17508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:01.111631 17508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:01.117034 17508 catalog_manager.cc:1347] Generated new cluster ID: f1fc3808323b421f96cf9a7537875ab9
I20260924 04:07:01.117107 17508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:01.137138 17508 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:01.138396 17508 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:01.146934 17508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac: Generated new TSK 0
I20260924 04:07:01.147867 17508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:07:01.176561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35731
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:01.290336 17524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:01.290634 17524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:01.290680 17524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:01.294916 17524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:01.294991 17524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:01.295058 17524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:07:01.300482 17524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:01.301539 17524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:01.302522 17524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:01.312153 17532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:01.312196 17529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:01.312214 17524 server_base.cc:1034] running on GCE node
W20260924 04:07:01.312156 17530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:01.312984 17524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:01.313616 17524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:01.314776 17524 hybrid_clock.cc:648] HybridClock initialized: now 1790222821314756 us; error 33 us; skew 500 ppm
Sep 24 04:07:01 dist-test-slave-zcsk krb5kdc[17417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:01.317400 17524 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:07:01.318251 17524 webserver.cc:460] Webserver started at http://127.0.98.1:39571/ using document root <none> and password file <none>
I20260924 04:07:01.318812 17524 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:01.318876 17524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:01.319068 17524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:01.321115 17524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "472cf32ea18348b8be231bd9f781e0c8"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "112eb12414eec3166eba3b8ba769d7c6"
server_key_iv: "3b209766f306332b1f656e90cc05055c"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.321643 17524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "472cf32ea18348b8be231bd9f781e0c8"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "112eb12414eec3166eba3b8ba769d7c6"
server_key_iv: "3b209766f306332b1f656e90cc05055c"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.325354 17524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 04:07:01.327745 17539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.328986 17524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:07:01.329123 17524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "472cf32ea18348b8be231bd9f781e0c8"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "112eb12414eec3166eba3b8ba769d7c6"
server_key_iv: "3b209766f306332b1f656e90cc05055c"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.329219 17524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:01.351435 17524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:01.354856 17524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:01.355026 17524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:01.355572 17524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:01.356509 17524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:01.356559 17524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.356609 17524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:01.356624 17524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.365283 17524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35579
I20260924 04:07:01.365356 17652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35579 every 8 connection(s)
I20260924 04:07:01.366250 17524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:07:01.373420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 17524
I20260924 04:07:01.373572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:07:01.373904 392 external_mini_cluster.cc:1428] Setting key 3b049b0e3ec4e93c449011a18d43fdec
Sep 24 04:07:01 dist-test-slave-zcsk krb5kdc[17417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:01.380970 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.368068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54495 (local address 127.0.98.62:35731)
0924 04:07:01.368198 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.368202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.369109 (+ 907us) server_negotiation.cc:407] Connection header received
0924 04:07:01.370419 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.370422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.370474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.370560 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:01.371788 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.372403 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.374054 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.374330 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.374453 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.377407 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:01.377410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:01.377412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:01.377437 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:01.378818 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.379382 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.379385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.379386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.379467 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.379800 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.379803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.379805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.380031 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:01.380120 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.380666 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.380802 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:07:01.381844 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.368455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35731 (local address 127.0.98.1:54495)
0924 04:07:01.368937 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:01.368984 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:01.370140 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:01.370731 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:01.370739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:01.371113 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:01.371649 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.371663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.372555 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.372558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.373857 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.373870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.374510 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.374515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.374729 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.375386 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:01.375405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:01.377220 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:01.378936 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:01.378942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:01.378961 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:01.379282 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:01.379550 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:01.379553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:01.379555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:01.379704 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:01.380130 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:01.380138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:01.380427 (+ 289us) client_negotiation.cc:769] Sending connection context
0924 04:07:01.380607 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 04:07:01.380980 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:07:01.383250 17653 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35731
I20260924 04:07:01.383543 17653 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:01.384122 17653 heartbeater.cc:511] Master 127.0.98.62:35731 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:07:01.385803 17462 ts_manager.cc:194] Registered new tserver with Master: 472cf32ea18348b8be231bd9f781e0c8 (127.0.98.1:35579)
I20260924 04:07:01.387228 17462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:07:01.398650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35731
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:01.509591 17658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:01.509866 17658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:01.509912 17658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:01.514143 17658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:01.514214 17658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:01.514278 17658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:07:01.519768 17658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:01.520881 17658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:01.521950 17658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:01.531957 17664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:01.531955 17666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:01.531955 17663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:01.532133 17658 server_base.cc:1034] running on GCE node
I20260924 04:07:01.532740 17658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:01.533375 17658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:01.534535 17658 hybrid_clock.cc:648] HybridClock initialized: now 1790222821534516 us; error 30 us; skew 500 ppm
Sep 24 04:07:01 dist-test-slave-zcsk krb5kdc[17417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:01.537536 17658 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:07:01.538424 17658 webserver.cc:460] Webserver started at http://127.0.98.2:34705/ using document root <none> and password file <none>
I20260924 04:07:01.538978 17658 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:01.539034 17658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:01.539223 17658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:01.541427 17658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "7a4eb50d8754455881dc893cf36b8152"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "381d7ffaa053f58ed1e3a89c2ffc4fd7"
server_key_iv: "8db01cc67b16349ee142e028b3f4d476"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.541976 17658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7a4eb50d8754455881dc893cf36b8152"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "381d7ffaa053f58ed1e3a89c2ffc4fd7"
server_key_iv: "8db01cc67b16349ee142e028b3f4d476"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.545785 17658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 04:07:01.548355 17673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.549594 17658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:01.549713 17658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "7a4eb50d8754455881dc893cf36b8152"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "381d7ffaa053f58ed1e3a89c2ffc4fd7"
server_key_iv: "8db01cc67b16349ee142e028b3f4d476"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.549824 17658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:01.572896 17658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:01.576191 17658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:01.576380 17658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:01.577010 17658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:01.577853 17658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:01.577901 17658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.577953 17658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:01.577982 17658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.586620 17658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41363
I20260924 04:07:01.586639 17786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41363 every 8 connection(s)
I20260924 04:07:01.587546 17658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:07:01.595284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 17658
I20260924 04:07:01.595458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:07:01.595834 392 external_mini_cluster.cc:1428] Setting key 123755d08a79dfa4fbc982b605d665fd
Sep 24 04:07:01 dist-test-slave-zcsk krb5kdc[17417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:01.601945 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.589403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54727 (local address 127.0.98.62:35731)
0924 04:07:01.589530 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.589534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.590403 (+ 869us) server_negotiation.cc:407] Connection header received
0924 04:07:01.591703 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.591707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.591770 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.591851 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:01.593136 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.593675 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.594849 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.595097 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.595194 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.598131 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:01.598134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:01.598136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:01.598161 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:01.599681 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.600311 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.600314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.600316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.600407 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.600752 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.600755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.600756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.600949 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:01.601036 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.601635 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.601774 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:07:01.602753 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.589765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35731 (local address 127.0.98.2:54727)
0924 04:07:01.590231 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:01.590278 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:01.591467 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:01.592066 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:01.592074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:01.592460 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:01.592980 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.592995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.593806 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.593809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.594726 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.594736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.595248 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.595252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.595431 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.596079 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:01.596097 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:01.597925 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:01.599866 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:01.599871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:01.599887 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:01.600213 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:01.600480 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:01.600483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:01.600484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:01.600619 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:01.601054 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:01.601062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:01.601350 (+ 288us) client_negotiation.cc:769] Sending connection context
0924 04:07:01.601529 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 04:07:01.601850 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260924 04:07:01.604045 17787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35731
I20260924 04:07:01.604336 17787 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:01.604977 17787 heartbeater.cc:511] Master 127.0.98.62:35731 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:07:01.606331 17462 ts_manager.cc:194] Registered new tserver with Master: 7a4eb50d8754455881dc893cf36b8152 (127.0.98.2:41363)
I20260924 04:07:01.607040 17462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:07:01.621296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35731
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:01.742889 17792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:01.743306 17792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:01.743391 17792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:01.749752 17792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:01.749843 17792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:01.749931 17792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:07:01.755810 17792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:01.756953 17792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:01.758029 17792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:01.768250 17797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:01.768255 17800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:01.768555 17792 server_base.cc:1034] running on GCE node
W20260924 04:07:01.768255 17798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:01.769132 17792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:01.770073 17792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:01.771278 17792 hybrid_clock.cc:648] HybridClock initialized: now 1790222821771239 us; error 58 us; skew 500 ppm
Sep 24 04:07:01 dist-test-slave-zcsk krb5kdc[17417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:01.774663 17792 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:07:01.775720 17792 webserver.cc:460] Webserver started at http://127.0.98.3:43281/ using document root <none> and password file <none>
I20260924 04:07:01.776348 17792 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:01.776428 17792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:01.776655 17792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:01.778808 17792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "f597f6cd201f4e419bc456cb091070a3"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "f657193e0264275671821c11eee4cbdc"
server_key_iv: "5d963bc0ea8d7615e3d1bd8b6366725a"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.779387 17792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f597f6cd201f4e419bc456cb091070a3"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "f657193e0264275671821c11eee4cbdc"
server_key_iv: "5d963bc0ea8d7615e3d1bd8b6366725a"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.783269 17792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:07:01.786058 17807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.787483 17792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:07:01.787657 17792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "f597f6cd201f4e419bc456cb091070a3"
format_stamp: "Formatted at 2026-09-24 04:07:01 on dist-test-slave-zcsk"
server_key: "f657193e0264275671821c11eee4cbdc"
server_key_iv: "5d963bc0ea8d7615e3d1bd8b6366725a"
server_key_version: "encryptionkey@0"
I20260924 04:07:01.787778 17792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:01.801115 17792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:01.804380 17792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:01.804548 17792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:01.805099 17792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:01.805955 17792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:01.806005 17792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.806056 17792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:01.806085 17792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:01.814700 17792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46041
I20260924 04:07:01.814735 17920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46041 every 8 connection(s)
I20260924 04:07:01.815580 17792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:07:01.817627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 17792
I20260924 04:07:01.817740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:07:01.817996 392 external_mini_cluster.cc:1428] Setting key dc7d3314284e0d7c5ba8363bc4cee1f6
Sep 24 04:07:01 dist-test-slave-zcsk krb5kdc[17417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:01.828897 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.817364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35719 (local address 127.0.98.62:35731)
0924 04:07:01.817512 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.817517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.818429 (+ 912us) server_negotiation.cc:407] Connection header received
0924 04:07:01.819744 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.819748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.819802 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.819870 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:01.821082 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.821602 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.822761 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.822978 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.823064 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.825558 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:01.825561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:01.825563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:01.825589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:01.826799 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.827342 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.827346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.827347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.827427 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.827762 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.827766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.827767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.828001 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:01.828089 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.828601 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.828734 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:07:01.829785 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.817788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35731 (local address 127.0.98.3:35719)
0924 04:07:01.818276 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:01.818330 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:01.819536 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:01.820081 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:01.820089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:01.820451 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:01.820939 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.820952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.821729 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.821732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.822638 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.822647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.823116 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.823119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.823295 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.823946 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:01.823964 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:01.825393 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:01.826923 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:01.826929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:01.826946 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:01.827246 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:01.827516 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:01.827519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:01.827521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:01.827672 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:01.828097 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:01.828104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:01.828365 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 04:07:01.828542 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 04:07:01.828895 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260924 04:07:01.831152 17921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35731
I20260924 04:07:01.831432 17921 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:01.832075 17921 heartbeater.cc:511] Master 127.0.98.62:35731 requested a full tablet report, sending...
I20260924 04:07:01.833115 17461 ts_manager.cc:194] Registered new tserver with Master: f597f6cd201f4e419bc456cb091070a3 (127.0.98.3:46041)
I20260924 04:07:01.833814 17461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35719
I20260924 04:07:01.844003 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:07:01.853295 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.845986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.0.98.62:35731)
0924 04:07:01.846114 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.846118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.846279 (+ 161us) server_negotiation.cc:407] Connection header received
0924 04:07:01.846422 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.846425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.846470 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.846532 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:01.847305 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.847823 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.848736 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.848955 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.849044 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.850383 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:01.850386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:01.850389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:01.850416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:01.851580 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.852010 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.852014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.852016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.852096 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:01.852377 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:01.852380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:01.852382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:01.852611 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:01.852756 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.852986 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.853099 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:07:01.856583 17461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:07:01.859784 17461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:07:01.874897 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.868756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46041 (local address 127.0.0.1:40856)
0924 04:07:01.869104 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:01.869173 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:01.869317 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:01.870256 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:01.870262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:01.870299 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:01.870504 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.870514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.872058 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.872061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.873298 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.873308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.874241 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.874245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.874371 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.874427 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 04:07:01.874566 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 04:07:01.874715 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":84,"spinlock_wait_cycles":9728,"thread_start_us":91,"threads_started":1}
I20260924 04:07:01.874922 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.869323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.98.3:46041)
0924 04:07:01.869716 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.869728 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.869753 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:07:01.869833 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.869838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.870064 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.870483 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:01.870636 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.871871 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.873421 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.874116 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.874224 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.874378 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.874615 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.874727 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":178,"threads_started":1}
I20260924 04:07:01.875380 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.869984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41363 (local address 127.0.0.1:36938)
0924 04:07:01.870193 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:01.870207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:01.870303 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:01.871158 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:01.871161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:01.871178 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:01.871273 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.871278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.872451 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.872454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.873678 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.873688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.874874 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.874877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.874992 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.875099 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 04:07:01.875166 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:07:01.875242 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":16,"spinlock_wait_cycles":4736,"thread_start_us":85,"threads_started":1}
I20260924 04:07:01.875451 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.868740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.0.98.1:35579)
0924 04:07:01.869954 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.869969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.869988 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:07:01.870052 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.870057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.870221 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.870632 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:01.870783 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.871899 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.873835 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.874502 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.874979 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.875133 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.875192 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.875287 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":85,"threads_started":1}
I20260924 04:07:01.875769 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.868601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35579 (local address 127.0.0.1:59548)
0924 04:07:01.869104 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:01.869173 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:01.869317 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:01.870352 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:01.870355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:01.870368 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:01.870479 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.870486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.872057 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.872061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.873678 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.873688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.874646 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.874649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.874732 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.874753 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:07:01.874802 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 04:07:01.874860 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":155,"spinlock_wait_cycles":7040,"thread_start_us":95,"threads_started":1}
I20260924 04:07:01.876047 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.870398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.0.98.2:41363)
0924 04:07:01.870810 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.870818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.870834 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:07:01.870892 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.870895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.871034 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.871165 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:01.871364 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.872236 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.873805 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.874559 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.874665 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.874802 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.875756 (+ 954us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.875859 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":129,"threads_started":1}
I20260924 04:07:01.879549 17587 tablet_service.cc:1467] Processing CreateTablet for tablet b5391f50c0f84032995b6e18eff34674 (DEFAULT_TABLE table=test-table [id=27d90e2765ae46828793b75fcd697e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:01.879549 17855 tablet_service.cc:1467] Processing CreateTablet for tablet b5391f50c0f84032995b6e18eff34674 (DEFAULT_TABLE table=test-table [id=27d90e2765ae46828793b75fcd697e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:01.879547 17721 tablet_service.cc:1467] Processing CreateTablet for tablet b5391f50c0f84032995b6e18eff34674 (DEFAULT_TABLE table=test-table [id=27d90e2765ae46828793b75fcd697e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:01.880860 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5391f50c0f84032995b6e18eff34674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:01.880860 17855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5391f50c0f84032995b6e18eff34674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:01.880879 17721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5391f50c0f84032995b6e18eff34674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:01.887869 17938 tablet_bootstrap.cc:492] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152: Bootstrap starting.
I20260924 04:07:01.889233 17939 tablet_bootstrap.cc:492] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8: Bootstrap starting.
I20260924 04:07:01.889652 17940 tablet_bootstrap.cc:492] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: Bootstrap starting.
I20260924 04:07:01.890973 17938 tablet_bootstrap.cc:654] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:01.891393 17939 tablet_bootstrap.cc:654] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:01.891788 17940 tablet_bootstrap.cc:654] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:01.892134 17938 log.cc:824] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:01.892262 17939 log.cc:824] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:01.892980 17940 log.cc:824] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:01.894191 17938 tablet_bootstrap.cc:492] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152: No bootstrap required, opened a new log
I20260924 04:07:01.894268 17939 tablet_bootstrap.cc:492] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8: No bootstrap required, opened a new log
I20260924 04:07:01.894397 17938 ts_tablet_manager.cc:1397] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260924 04:07:01.894457 17939 ts_tablet_manager.cc:1397] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:07:01.894815 17940 tablet_bootstrap.cc:492] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: No bootstrap required, opened a new log
I20260924 04:07:01.894984 17940 ts_tablet_manager.cc:1397] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:07:01.897557 17938 raft_consensus.cc:348] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.898200 17938 raft_consensus.cc:374] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:01.898327 17938 raft_consensus.cc:729] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4eb50d8754455881dc893cf36b8152, State: Initialized, Role: FOLLOWER
I20260924 04:07:01.898970 17938 consensus_queue.cc:260] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.898910 17939 raft_consensus.cc:348] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.898910 17940 raft_consensus.cc:348] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.899211 17940 raft_consensus.cc:374] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:01.899240 17939 raft_consensus.cc:374] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:01.899277 17940 raft_consensus.cc:729] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f597f6cd201f4e419bc456cb091070a3, State: Initialized, Role: FOLLOWER
I20260924 04:07:01.899295 17939 raft_consensus.cc:729] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472cf32ea18348b8be231bd9f781e0c8, State: Initialized, Role: FOLLOWER
I20260924 04:07:01.899757 17939 consensus_queue.cc:260] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.899757 17940 consensus_queue.cc:260] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.900077 17938 ts_tablet_manager.cc:1428] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 04:07:01.900523 17787 heartbeater.cc:503] Master 127.0.98.62:35731 was elected leader, sending a full tablet report...
I20260924 04:07:01.900730 17921 heartbeater.cc:503] Master 127.0.98.62:35731 was elected leader, sending a full tablet report...
I20260924 04:07:01.900743 17653 heartbeater.cc:503] Master 127.0.98.62:35731 was elected leader, sending a full tablet report...
I20260924 04:07:01.900820 17940 ts_tablet_manager.cc:1428] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:07:01.901743 17939 ts_tablet_manager.cc:1428] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:07:01.953059 17945 raft_consensus.cc:482] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:07:01.953307 17945 raft_consensus.cc:504] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.954557 17945 leader_election.cc:288] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 472cf32ea18348b8be231bd9f781e0c8 (127.0.98.1:35579), 7a4eb50d8754455881dc893cf36b8152 (127.0.98.2:41363)
I20260924 04:07:01.958279 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.954806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35579 (local address 127.0.98.3:36901)
0924 04:07:01.954927 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:01.954943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:01.955040 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:01.955386 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:01.955389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:01.955409 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:01.955512 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.955517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.956411 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.956415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.957231 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.957240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.957863 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.957866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.957972 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.957990 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:07:01.958056 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:07:01.958134 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 04:07:01.958279 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.954955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36901 (local address 127.0.98.1:35579)
0924 04:07:01.955084 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.955089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.955106 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:07:01.955171 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.955175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.955225 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.955316 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:01.955614 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.956308 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.957353 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.957750 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.957840 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.957906 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.958085 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.958156 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:07:01.958492 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.955162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38489 (local address 127.0.98.2:41363)
0924 04:07:01.955311 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.955315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.955512 (+ 197us) server_negotiation.cc:407] Connection header received
0924 04:07:01.955669 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.955671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.955710 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.955761 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:01.956041 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.956523 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.957557 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.957968 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.958061 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.958121 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.958309 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.958388 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:07:01.958501 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.955070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41363 (local address 127.0.98.3:38489)
0924 04:07:01.955422 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:01.955435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:01.955573 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:01.955788 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:01.955791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:01.955808 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:01.955942 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.955949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.956633 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.956636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.957435 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:01.957443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.958082 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.958086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:01.958200 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.958219 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:07:01.958282 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:07:01.958355 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":165,"threads_started":1}
I20260924 04:07:01.959275 17741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5391f50c0f84032995b6e18eff34674" candidate_uuid: "f597f6cd201f4e419bc456cb091070a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a4eb50d8754455881dc893cf36b8152" is_pre_election: true
I20260924 04:07:01.959273 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5391f50c0f84032995b6e18eff34674" candidate_uuid: "f597f6cd201f4e419bc456cb091070a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472cf32ea18348b8be231bd9f781e0c8" is_pre_election: true
I20260924 04:07:01.959618 17741 raft_consensus.cc:2454] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f597f6cd201f4e419bc456cb091070a3 in term 0.
I20260924 04:07:01.959618 17607 raft_consensus.cc:2454] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f597f6cd201f4e419bc456cb091070a3 in term 0.
I20260924 04:07:01.960156 17808 leader_election.cc:302] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 472cf32ea18348b8be231bd9f781e0c8, f597f6cd201f4e419bc456cb091070a3; no voters:
I20260924 04:07:01.960417 17945 raft_consensus.cc:2781] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:07:01.960503 17945 raft_consensus.cc:482] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:07:01.960531 17945 raft_consensus.cc:3037] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:01.961565 17945 raft_consensus.cc:504] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.961913 17945 leader_election.cc:288] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [CANDIDATE]: Term 1 election: Requested vote from peers 472cf32ea18348b8be231bd9f781e0c8 (127.0.98.1:35579), 7a4eb50d8754455881dc893cf36b8152 (127.0.98.2:41363)
I20260924 04:07:01.962380 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5391f50c0f84032995b6e18eff34674" candidate_uuid: "f597f6cd201f4e419bc456cb091070a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472cf32ea18348b8be231bd9f781e0c8"
I20260924 04:07:01.962410 17741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5391f50c0f84032995b6e18eff34674" candidate_uuid: "f597f6cd201f4e419bc456cb091070a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a4eb50d8754455881dc893cf36b8152"
I20260924 04:07:01.962512 17607 raft_consensus.cc:3037] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:01.962524 17741 raft_consensus.cc:3037] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:01.963560 17741 raft_consensus.cc:2454] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f597f6cd201f4e419bc456cb091070a3 in term 1.
I20260924 04:07:01.963562 17607 raft_consensus.cc:2454] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f597f6cd201f4e419bc456cb091070a3 in term 1.
I20260924 04:07:01.963889 17808 leader_election.cc:302] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 472cf32ea18348b8be231bd9f781e0c8, f597f6cd201f4e419bc456cb091070a3; no voters:
I20260924 04:07:01.964103 17945 raft_consensus.cc:2781] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:01.964365 17945 raft_consensus.cc:686] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 1 LEADER]: Becoming Leader. State: Replica: f597f6cd201f4e419bc456cb091070a3, State: Running, Role: LEADER
I20260924 04:07:01.964696 17945 consensus_queue.cc:237] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } }
I20260924 04:07:01.968151 17460 catalog_manager.cc:5495] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f597f6cd201f4e419bc456cb091070a3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f597f6cd201f4e419bc456cb091070a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f597f6cd201f4e419bc456cb091070a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 } health_report { overall_health: UNKNOWN } } }
I20260924 04:07:01.998319 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:01.994778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.0.98.3:46041)
0924 04:07:01.994909 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:01.994912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:01.994926 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:07:01.995004 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:01.995006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:01.995062 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:01.995182 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:01.995394 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.995942 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.996860 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:01.997073 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:01.997156 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:01.997468 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:01.997912 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:01.998023 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:01.998112 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:07:01.998172 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:07:02.004642 17741 raft_consensus.cc:1261] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 1 FOLLOWER]: Refusing update from remote peer f597f6cd201f4e419bc456cb091070a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:02.004667 17607 raft_consensus.cc:1261] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8 [term 1 FOLLOWER]: Refusing update from remote peer f597f6cd201f4e419bc456cb091070a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:02.005422 17948 consensus_queue.cc:1035] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "472cf32ea18348b8be231bd9f781e0c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:02.005648 17945 consensus_queue.cc:1035] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a4eb50d8754455881dc893cf36b8152" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:02.016311 17953 mvcc.cc:205] Tried to move back new op lower bound from 7332752678922850304 to 7332752678770376704. Current Snapshot: MvccSnapshot[applied={T|T < 7332752678922850304}]
I20260924 04:07:02.027694 17460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:02.027879 17460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:02.030431 17460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c10660f8f90427283ff9aebdf8644ac: Sending DeleteTablet for 3 replicas of tablet b5391f50c0f84032995b6e18eff34674
I20260924 04:07:02.031267 17855 tablet_service.cc:1514] Processing DeleteTablet for tablet b5391f50c0f84032995b6e18eff34674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40856
I20260924 04:07:02.031268 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet b5391f50c0f84032995b6e18eff34674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59548
I20260924 04:07:02.031523 17721 tablet_service.cc:1514] Processing DeleteTablet for tablet b5391f50c0f84032995b6e18eff34674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36938
I20260924 04:07:02.031821 17961 tablet_replica.cc:330] T b5391f50c0f84032995b6e18eff34674 P 472cf32ea18348b8be231bd9f781e0c8: stopping tablet replica
I20260924 04:07:02.031955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 17524
I20260924 04:07:02.036159 17962 tablet_replica.cc:330] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: stopping tablet replica
I20260924 04:07:02.036554 17963 tablet_replica.cc:330] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152: stopping tablet replica
I20260924 04:07:02.036551 17962 raft_consensus.cc:2229] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:07:02.036979 17963 raft_consensus.cc:2229] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:02.037031 17962 raft_consensus.cc:2258] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:02.037397 17963 raft_consensus.cc:2258] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:07:02.038069 17446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35579 (error 108)
I20260924 04:07:02.038947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 17658
I20260924 04:07:02.039019 17962 ts_tablet_manager.cc:1905] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:02.039021 17963 ts_tablet_manager.cc:1905] T b5391f50c0f84032995b6e18eff34674 P 7a4eb50d8754455881dc893cf36b8152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:02.041870 17962 ts_tablet_manager.cc:1918] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:02.041971 17962 log.cc:1196] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/b5391f50c0f84032995b6e18eff34674
I20260924 04:07:02.042301 17962 ts_tablet_manager.cc:1939] T b5391f50c0f84032995b6e18eff34674 P f597f6cd201f4e419bc456cb091070a3: Deleting consensus metadata
I20260924 04:07:02.043705 17447 catalog_manager.cc:4841] TS f597f6cd201f4e419bc456cb091070a3 (127.0.98.3:46041): tablet b5391f50c0f84032995b6e18eff34674 (table test-table [id=27d90e2765ae46828793b75fcd697e53]) successfully deleted
I20260924 04:07:02.045043 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.044717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35579 (local address 127.0.0.1:59552)
0924 04:07:02.044875 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:02.044953 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 04:07:02.045239 17446 catalog_manager.cc:4584] TS 472cf32ea18348b8be231bd9f781e0c8 (127.0.98.1:35579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5391f50c0f84032995b6e18eff34674: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35579: connect: Connection refused (error 111)
I20260924 04:07:02.046175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 17792
I20260924 04:07:02.046510 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.046242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41363 (local address 127.0.0.1:36950)
0924 04:07:02.046388 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:02.046442 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 04:07:02.051440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 17430
2026-09-24T04:07:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2260 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T04:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:02Z Disabled control of system clock
I20260924 04:07:02.081148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36889
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:07:02.190599 17968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:02.190877 17968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:02.190912 17968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:02.194952 17968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:02.195034 17968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:02.195050 17968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:02.195065 17968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:02.195079 17968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:02.200014 17968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:02.201095 17968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:02.202454 17968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:02.210702 17976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:02.210695 17974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:02.210695 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:02.211009 17968 server_base.cc:1034] running on GCE node
I20260924 04:07:02.211647 17968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:02.212838 17968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:02.214000 17968 hybrid_clock.cc:648] HybridClock initialized: now 1790222822213973 us; error 42 us; skew 500 ppm
I20260924 04:07:02.215896 17968 webserver.cc:460] Webserver started at http://127.0.98.62:45207/ using document root <none> and password file <none>
I20260924 04:07:02.216521 17968 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:02.216586 17968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:02.216765 17968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:02.218835 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "900e813229364a01af238436b74dd8ef"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "f9199acb88d7a4c1876e88b201836bda"
server_key_iv: "4f09959edf9fca30a6e770eae2734e06"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.219352 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "900e813229364a01af238436b74dd8ef"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "f9199acb88d7a4c1876e88b201836bda"
server_key_iv: "4f09959edf9fca30a6e770eae2734e06"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.223102 17968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:07:02.225656 17982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.226971 17968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:07:02.227084 17968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "900e813229364a01af238436b74dd8ef"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "f9199acb88d7a4c1876e88b201836bda"
server_key_iv: "4f09959edf9fca30a6e770eae2734e06"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.227197 17968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:02.237390 17968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:02.238054 17968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:02.238216 17968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:02.244975 17968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36889
I20260924 04:07:02.245013 18034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36889 every 8 connection(s)
I20260924 04:07:02.245916 17968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:02.246860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 17968
I20260924 04:07:02.247005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:02.247309 392 external_mini_cluster.cc:1428] Setting key d333b0e1a2fd8eebad44a2982ba941f0
I20260924 04:07:02.249600 18035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:02.256232 18035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef: Bootstrap starting.
I20260924 04:07:02.257025 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.248752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.0.98.62:36889)
0924 04:07:02.249405 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:02.249432 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:02.249469 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:07:02.250188 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:02.250206 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:02.250623 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:02.250935 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:02.251279 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.252325 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.253534 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.254049 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.254171 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.255051 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:02.255056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:02.255070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:02.255532 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:02.255599 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:02.255620 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:02.255693 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:02.255858 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:07:02.256194 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260924 04:07:02.258769 18035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:02.259642 18035 log.cc:824] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:02.261925 18035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef: No bootstrap required, opened a new log
I20260924 04:07:02.265784 18035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900e813229364a01af238436b74dd8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260924 04:07:02.266074 18035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:02.266141 18035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900e813229364a01af238436b74dd8ef, State: Initialized, Role: FOLLOWER
I20260924 04:07:02.266605 18035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900e813229364a01af238436b74dd8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260924 04:07:02.266738 18035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:02.266805 18035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:02.266906 18035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:02.267854 18035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900e813229364a01af238436b74dd8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260924 04:07:02.268218 18035 leader_election.cc:302] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 900e813229364a01af238436b74dd8ef; no voters:
I20260924 04:07:02.268556 18035 leader_election.cc:288] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:02.268671 18040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:02.268898 18040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [term 1 LEADER]: Becoming Leader. State: Replica: 900e813229364a01af238436b74dd8ef, State: Running, Role: LEADER
I20260924 04:07:02.269295 18040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900e813229364a01af238436b74dd8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260924 04:07:02.269804 18035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:02.270927 18041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "900e813229364a01af238436b74dd8ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900e813229364a01af238436b74dd8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } } }
I20260924 04:07:02.270915 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 900e813229364a01af238436b74dd8ef. Latest consensus state: current_term: 1 leader_uuid: "900e813229364a01af238436b74dd8ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900e813229364a01af238436b74dd8ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } } }
I20260924 04:07:02.271057 18041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:02.271057 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:02.271585 18049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:02.274708 18049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:02.281023 18049 catalog_manager.cc:1347] Generated new cluster ID: 307e78382d5c46728b1598d97e9daed0
I20260924 04:07:02.281126 18049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:02.323096 18049 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:02.324138 18049 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:02.332252 18049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef: Generated new TSK 0
I20260924 04:07:02.333177 18049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:02.350543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:07:02.462972 18059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:02.463274 18059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:02.463328 18059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:02.467414 18059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:02.467489 18059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:02.467567 18059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:07:02.472683 18059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:02.473845 18059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:02.475234 18059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:02.484869 18065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:02.484861 18067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:02.484864 18064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:02.485286 18059 server_base.cc:1034] running on GCE node
I20260924 04:07:02.485783 18059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:02.486652 18059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:02.487874 18059 hybrid_clock.cc:648] HybridClock initialized: now 1790222822487829 us; error 62 us; skew 500 ppm
I20260924 04:07:02.489820 18059 webserver.cc:460] Webserver started at http://127.0.98.1:34685/ using document root <none> and password file <none>
I20260924 04:07:02.490386 18059 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:02.490456 18059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:02.490655 18059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:02.492843 18059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "98d058724c144509a2c795dfc10d9347"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "ab5764ca11ffd04128fc1431065a7658"
server_key_iv: "37b425256cbbf6cddf67ece78b4abf7a"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.493377 18059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "98d058724c144509a2c795dfc10d9347"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "ab5764ca11ffd04128fc1431065a7658"
server_key_iv: "37b425256cbbf6cddf67ece78b4abf7a"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.497135 18059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:07:02.499837 18073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.501197 18059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:02.501302 18059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "98d058724c144509a2c795dfc10d9347"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "ab5764ca11ffd04128fc1431065a7658"
server_key_iv: "37b425256cbbf6cddf67ece78b4abf7a"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.501422 18059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:02.515324 18059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:02.516122 18059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:02.516301 18059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:02.516947 18059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:02.517935 18059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:02.517982 18059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.518023 18059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:02.518055 18059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.526852 18059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39255
I20260924 04:07:02.526949 18186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39255 every 8 connection(s)
I20260924 04:07:02.527812 18059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:07:02.537230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 18059
I20260924 04:07:02.537367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:07:02.537597 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.529727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45685 (local address 127.0.98.62:36889)
0924 04:07:02.529872 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:02.529876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:02.531127 (+ 1251us) server_negotiation.cc:407] Connection header received
0924 04:07:02.532676 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:02.532680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:02.532748 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:02.532823 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:02.533771 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.534316 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.535483 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.535716 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.535815 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.536949 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:02.536953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:02.536956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:02.536972 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:02.536997 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:02.537009 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:02.537089 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:02.537282 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 04:07:02.537422 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:07:02.537715 392 external_mini_cluster.cc:1428] Setting key 817d4ee03bd5fa6b02d63e1b2c705c72
I20260924 04:07:02.538506 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.530213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.1:45685)
0924 04:07:02.530946 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:02.530993 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:02.532449 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:02.533031 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:02.533039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:02.533130 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:02.533627 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.533641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.534442 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.534447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.535359 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.535370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.535860 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.535876 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.536090 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.536666 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:02.536668 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:02.536710 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:02.536779 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:02.537125 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:02.537134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:02.537139 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:07:02.537253 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:07:02.537551 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":118,"threads_started":1}
I20260924 04:07:02.539839 18187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260924 04:07:02.540012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:07:02.540131 18187 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:02.540683 18187 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260924 04:07:02.542323 17999 ts_manager.cc:194] Registered new tserver with Master: 98d058724c144509a2c795dfc10d9347 (127.0.98.1:39255)
I20260924 04:07:02.543442 17999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45685
W20260924 04:07:02.652422 18190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:02.652712 18190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:02.652753 18190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:02.656836 18190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:02.656889 18190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:02.656947 18190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:07:02.661716 18190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:02.662758 18190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:02.664129 18190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:02.674022 18195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:02.674067 18190 server_base.cc:1034] running on GCE node
W20260924 04:07:02.674029 18198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:02.674039 18196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:02.674677 18190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:02.675326 18190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:02.676506 18190 hybrid_clock.cc:648] HybridClock initialized: now 1790222822676484 us; error 31 us; skew 500 ppm
I20260924 04:07:02.678320 18190 webserver.cc:460] Webserver started at http://127.0.98.2:42241/ using document root <none> and password file <none>
I20260924 04:07:02.678898 18190 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:02.678962 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:02.679160 18190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:02.681282 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "f426e657edad44ab856b0646478cc8cb"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "eab5b88940943b9516026f5ce97b0e38"
server_key_iv: "b353f47ebb5de01b2a1bad6241527df0"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.681819 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f426e657edad44ab856b0646478cc8cb"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "eab5b88940943b9516026f5ce97b0e38"
server_key_iv: "b353f47ebb5de01b2a1bad6241527df0"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.685508 18190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:07:02.688001 18204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.689317 18190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:07:02.689440 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "f426e657edad44ab856b0646478cc8cb"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "eab5b88940943b9516026f5ce97b0e38"
server_key_iv: "b353f47ebb5de01b2a1bad6241527df0"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.689555 18190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:02.701150 18190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:02.701870 18190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:02.702042 18190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:02.702639 18190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:02.703603 18190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:02.703650 18190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.703693 18190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:02.703722 18190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.712150 18190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35001
I20260924 04:07:02.712180 18317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35001 every 8 connection(s)
I20260924 04:07:02.713122 18190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:07:02.717825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 18190
I20260924 04:07:02.717938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:07:02.718199 392 external_mini_cluster.cc:1428] Setting key c09f92a36abe11bf3c284576c3512412
I20260924 04:07:02.720360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:07:02.723098 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.714957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50623 (local address 127.0.98.62:36889)
0924 04:07:02.715098 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:02.715102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:02.716242 (+ 1140us) server_negotiation.cc:407] Connection header received
0924 04:07:02.717653 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:02.717657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:02.717720 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:02.717792 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:02.718843 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.719548 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.721190 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.721405 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.721483 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.722641 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:02.722645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:02.722648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:02.722663 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:02.722688 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:02.722699 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:02.722771 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:02.722879 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 04:07:02.722949 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:07:02.724047 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.715351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.2:50623)
0924 04:07:02.716096 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:02.716139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:02.717411 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:02.717989 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:02.717999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:02.718106 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:02.718694 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.718716 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.719695 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.719698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.721073 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.721086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.721548 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.721552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.721760 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.722371 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:02.722372 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:02.722416 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:02.722485 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:02.722769 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:02.722774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:02.722777 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:07:02.722868 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:07:02.723124 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20260924 04:07:02.725247 18318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260924 04:07:02.725531 18318 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:02.726241 18318 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260924 04:07:02.727456 17999 ts_manager.cc:194] Registered new tserver with Master: f426e657edad44ab856b0646478cc8cb (127.0.98.2:35001)
I20260924 04:07:02.728085 17999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50623
W20260924 04:07:02.835016 18321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:02.835299 18321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:02.835340 18321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:02.839516 18321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:02.839576 18321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:02.839640 18321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:07:02.844537 18321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:02.845621 18321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:02.847041 18321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:02.856626 18326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:02.856626 18329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:02.856619 18327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:02.856896 18321 server_base.cc:1034] running on GCE node
I20260924 04:07:02.857425 18321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:02.858083 18321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:02.859251 18321 hybrid_clock.cc:648] HybridClock initialized: now 1790222822859229 us; error 34 us; skew 500 ppm
I20260924 04:07:02.861261 18321 webserver.cc:460] Webserver started at http://127.0.98.3:35297/ using document root <none> and password file <none>
I20260924 04:07:02.861923 18321 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:02.862000 18321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:02.862213 18321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:02.864451 18321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc5dd3bb2e164bf8a061d802c451e5fa"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "74fb7bf7166de2610a277e8533550ff6"
server_key_iv: "eb32c5ef247a29ab859ef3597fd83549"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.865000 18321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc5dd3bb2e164bf8a061d802c451e5fa"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "74fb7bf7166de2610a277e8533550ff6"
server_key_iv: "eb32c5ef247a29ab859ef3597fd83549"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.868891 18321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:07:02.871570 18335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.872977 18321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:07:02.873103 18321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "fc5dd3bb2e164bf8a061d802c451e5fa"
format_stamp: "Formatted at 2026-09-24 04:07:02 on dist-test-slave-zcsk"
server_key: "74fb7bf7166de2610a277e8533550ff6"
server_key_iv: "eb32c5ef247a29ab859ef3597fd83549"
server_key_version: "encryptionkey@0"
I20260924 04:07:02.873222 18321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:02.884194 18321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:02.884899 18321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:02.885073 18321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:02.885667 18321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:02.886655 18321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:02.886703 18321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.886746 18321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:02.886776 18321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:02.895051 18321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45371
I20260924 04:07:02.895136 18448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45371 every 8 connection(s)
I20260924 04:07:02.896001 18321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:07:02.896745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 18321
I20260924 04:07:02.896835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:07:02.897101 392 external_mini_cluster.cc:1428] Setting key 5ed151dd3c47c84b200d54af197f25dc
I20260924 04:07:02.904933 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.897875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37487 (local address 127.0.98.62:36889)
0924 04:07:02.898040 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:02.898045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:02.899334 (+ 1289us) server_negotiation.cc:407] Connection header received
0924 04:07:02.900687 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:02.900690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:02.900736 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:02.900804 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:02.901646 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.902160 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.903258 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.903465 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.903535 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.904505 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:02.904508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:02.904510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:02.904522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:02.904541 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:02.904574 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:02.904627 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:02.904741 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:07:02.904789 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:07:02.905834 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.898415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.3:37487)
0924 04:07:02.899188 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:02.899243 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:02.900491 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:02.900936 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:02.900943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:02.901030 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:02.901530 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.901543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.902246 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.902249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.903165 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.903174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.903595 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.903597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.903734 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.904274 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:02.904276 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:02.904316 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:02.904379 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:02.904635 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:02.904641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:02.904645 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:07:02.904735 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:07:02.905023 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260924 04:07:02.906893 18449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260924 04:07:02.907130 18449 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:02.907622 18449 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260924 04:07:02.908646 17999 ts_manager.cc:194] Registered new tserver with Master: fc5dd3bb2e164bf8a061d802c451e5fa (127.0.98.3:45371)
I20260924 04:07:02.909207 17999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37487
I20260924 04:07:02.911048 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:07:02.916567 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.913025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.0.98.62:36889)
0924 04:07:02.913147 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:02.913150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:02.913386 (+ 236us) server_negotiation.cc:407] Connection header received
0924 04:07:02.913434 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:02.913436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:02.913481 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:02.913557 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:02.913808 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.914318 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.915221 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.915433 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.915510 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.916235 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:02.916238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:02.916240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:02.916250 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:02.916268 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:02.916275 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:02.916329 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:02.916406 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 04:07:02.916454 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:07:02.919255 17999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:07:02.922302 17999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:07:02.936539 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.930610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35001 (local address 127.0.0.1:35224)
0924 04:07:02.930948 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:02.930995 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:02.931272 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:02.932118 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:02.932124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:02.932150 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:02.932279 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.932286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.933684 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.933687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.934862 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.934870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.935995 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.935999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.936118 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.936161 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:07:02.936263 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:07:02.936385 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260924 04:07:02.936646 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.930745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.98.2:35001)
0924 04:07:02.931092 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:02.931105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:02.931131 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:07:02.931544 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:02.931551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:02.931767 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:02.931948 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:02.932406 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.933575 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.934989 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.935832 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.935996 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.936213 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:02.936309 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 04:07:02.936474 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":147,"threads_started":1}
I20260924 04:07:02.936707 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.931246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39255 (local address 127.0.0.1:56674)
0924 04:07:02.931511 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:02.931524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:02.931612 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:02.932747 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:02.932749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:02.932763 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:02.932863 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.932870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.934058 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.934062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.935277 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.935288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.936359 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.936363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.936452 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.936475 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:07:02.936525 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 04:07:02.936592 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260924 04:07:02.937105 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.931336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45371 (local address 127.0.0.1:44642)
0924 04:07:02.932420 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:02.932432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:02.932519 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:02.933048 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:02.933050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:02.933064 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:02.933118 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.933121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.934213 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.934216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.935392 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:02.935400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.936480 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.936483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:02.936577 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.936599 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:07:02.936916 (+ 317us) client_negotiation.cc:240] Negotiation successful
0924 04:07:02.936977 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":90,"threads_started":1}
I20260924 04:07:02.937515 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.931993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.0.98.1:39255)
0924 04:07:02.932313 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:02.932325 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:02.932348 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:07:02.932421 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:02.932426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:02.932619 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:02.932772 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:02.932946 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.933948 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.935411 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.936227 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.937066 (+ 839us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.937245 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:02.937294 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:07:02.937385 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260924 04:07:02.937582 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:02.931727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.0.98.3:45371)
0924 04:07:02.932065 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:02.932077 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:02.932504 (+ 427us) server_negotiation.cc:407] Connection header received
0924 04:07:02.932617 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:02.932621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:02.932788 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:02.932933 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:02.933215 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.934101 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.935501 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:02.936147 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:02.936248 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:02.936422 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:02.937315 (+ 893us) server_negotiation.cc:299] Negotiation successful
0924 04:07:02.937415 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":97,"threads_started":1}
I20260924 04:07:02.939999 18121 tablet_service.cc:1467] Processing CreateTablet for tablet 45f31820fe564adf997cdee867c138dc (DEFAULT_TABLE table=test-table [id=d9849f78aceb47f68c2283964c90bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:02.940012 18252 tablet_service.cc:1467] Processing CreateTablet for tablet 45f31820fe564adf997cdee867c138dc (DEFAULT_TABLE table=test-table [id=d9849f78aceb47f68c2283964c90bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:02.940012 18383 tablet_service.cc:1467] Processing CreateTablet for tablet 45f31820fe564adf997cdee867c138dc (DEFAULT_TABLE table=test-table [id=d9849f78aceb47f68c2283964c90bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:02.940935 18121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f31820fe564adf997cdee867c138dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:02.940972 18252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f31820fe564adf997cdee867c138dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:02.940972 18383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f31820fe564adf997cdee867c138dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:02.948328 18466 tablet_bootstrap.cc:492] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347: Bootstrap starting.
I20260924 04:07:02.948326 18467 tablet_bootstrap.cc:492] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Bootstrap starting.
I20260924 04:07:02.948798 18468 tablet_bootstrap.cc:492] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Bootstrap starting.
I20260924 04:07:02.950609 18468 tablet_bootstrap.cc:654] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:02.950671 18467 tablet_bootstrap.cc:654] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:02.950671 18466 tablet_bootstrap.cc:654] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:02.951373 18468 log.cc:824] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:02.951512 18466 log.cc:824] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:02.951512 18467 log.cc:824] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:02.953481 18466 tablet_bootstrap.cc:492] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347: No bootstrap required, opened a new log
I20260924 04:07:02.953481 18467 tablet_bootstrap.cc:492] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: No bootstrap required, opened a new log
I20260924 04:07:02.953481 18468 tablet_bootstrap.cc:492] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: No bootstrap required, opened a new log
I20260924 04:07:02.953704 18466 ts_tablet_manager.cc:1397] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:07:02.953706 18468 ts_tablet_manager.cc:1397] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:07:02.953729 18467 ts_tablet_manager.cc:1397] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:07:02.957031 18468 raft_consensus.cc:348] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:02.957301 18468 raft_consensus.cc:374] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:02.957371 18468 raft_consensus.cc:729] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5dd3bb2e164bf8a061d802c451e5fa, State: Initialized, Role: FOLLOWER
I20260924 04:07:02.957824 18468 consensus_queue.cc:260] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:02.958033 18467 raft_consensus.cc:348] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:02.958033 18466 raft_consensus.cc:348] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:02.958309 18467 raft_consensus.cc:374] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:02.958309 18466 raft_consensus.cc:374] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:02.958366 18467 raft_consensus.cc:729] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f426e657edad44ab856b0646478cc8cb, State: Initialized, Role: FOLLOWER
I20260924 04:07:02.958366 18466 raft_consensus.cc:729] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d058724c144509a2c795dfc10d9347, State: Initialized, Role: FOLLOWER
I20260924 04:07:02.958640 18449 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
I20260924 04:07:02.958824 18467 consensus_queue.cc:260] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:02.958824 18466 consensus_queue.cc:260] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:02.958904 18468 ts_tablet_manager.cc:1428] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:07:02.959704 18318 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
I20260924 04:07:02.959810 18467 ts_tablet_manager.cc:1428] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:07:02.959810 18466 ts_tablet_manager.cc:1428] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:07:02.960760 18187 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
W20260924 04:07:02.964075 18319 tablet.cc:2369] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:07:03.028972 18188 tablet.cc:2369] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:07:03.124645 18473 raft_consensus.cc:482] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:07:03.124886 18473 raft_consensus.cc:504] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:03.126035 18473 leader_election.cc:288] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f426e657edad44ab856b0646478cc8cb (127.0.98.2:35001), fc5dd3bb2e164bf8a061d802c451e5fa (127.0.98.3:45371)
I20260924 04:07:03.129884 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:03.126417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36037 (local address 127.0.98.2:35001)
0924 04:07:03.126558 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:03.126563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:03.126683 (+ 120us) server_negotiation.cc:407] Connection header received
0924 04:07:03.126857 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:03.126860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:03.126914 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:03.126962 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:03.127362 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.127872 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.128973 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.129343 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.129436 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:03.129496 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:03.129657 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:07:03.129733 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:07:03.129904 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:03.126262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35001 (local address 127.0.98.1:36037)
0924 04:07:03.126557 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:03.126574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:03.126713 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:03.127104 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:03.127107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:03.127125 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:03.127249 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:03.127256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.127983 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.127987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:03.128855 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:03.128864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.129457 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.129460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:03.129546 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:03.129564 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:07:03.129628 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:07:03.129712 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":45,"thread_start_us":110,"threads_started":1}
I20260924 04:07:03.129930 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:03.126799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45049 (local address 127.0.98.3:45371)
0924 04:07:03.126939 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:03.126944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:03.126957 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:07:03.127045 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:03.127048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:03.127091 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:03.127164 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:03.127490 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.128016 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.129008 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.129410 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.129498 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:03.129556 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:03.129740 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:07:03.129812 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:07:03.129951 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:03.126638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45371 (local address 127.0.98.1:45049)
0924 04:07:03.126881 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:03.126893 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:03.126967 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:03.127211 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:03.127214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:03.127230 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:03.127350 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:03.127355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.128120 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.128123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:03.128893 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:03.128901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.129525 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.129528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:03.129619 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:03.129642 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:07:03.129707 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:07:03.129770 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":4,"thread_start_us":74,"threads_started":1}
I20260924 04:07:03.130775 18272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f31820fe564adf997cdee867c138dc" candidate_uuid: "98d058724c144509a2c795dfc10d9347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f426e657edad44ab856b0646478cc8cb" is_pre_election: true
I20260924 04:07:03.130771 18403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f31820fe564adf997cdee867c138dc" candidate_uuid: "98d058724c144509a2c795dfc10d9347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" is_pre_election: true
I20260924 04:07:03.131106 18403 raft_consensus.cc:2454] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98d058724c144509a2c795dfc10d9347 in term 0.
I20260924 04:07:03.131106 18272 raft_consensus.cc:2454] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98d058724c144509a2c795dfc10d9347 in term 0.
I20260924 04:07:03.131654 18074 leader_election.cc:302] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98d058724c144509a2c795dfc10d9347, f426e657edad44ab856b0646478cc8cb; no voters:
I20260924 04:07:03.131989 18473 raft_consensus.cc:2781] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:07:03.132083 18473 raft_consensus.cc:482] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:07:03.132115 18473 raft_consensus.cc:3037] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:03.133100 18473 raft_consensus.cc:504] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:03.133420 18473 leader_election.cc:288] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [CANDIDATE]: Term 1 election: Requested vote from peers f426e657edad44ab856b0646478cc8cb (127.0.98.2:35001), fc5dd3bb2e164bf8a061d802c451e5fa (127.0.98.3:45371)
I20260924 04:07:03.133803 18272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f31820fe564adf997cdee867c138dc" candidate_uuid: "98d058724c144509a2c795dfc10d9347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f426e657edad44ab856b0646478cc8cb"
I20260924 04:07:03.133899 18403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f31820fe564adf997cdee867c138dc" candidate_uuid: "98d058724c144509a2c795dfc10d9347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa"
I20260924 04:07:03.133942 18272 raft_consensus.cc:3037] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:03.134055 18403 raft_consensus.cc:3037] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:03.134981 18272 raft_consensus.cc:2454] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98d058724c144509a2c795dfc10d9347 in term 1.
I20260924 04:07:03.135066 18403 raft_consensus.cc:2454] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98d058724c144509a2c795dfc10d9347 in term 1.
I20260924 04:07:03.135267 18074 leader_election.cc:302] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98d058724c144509a2c795dfc10d9347, f426e657edad44ab856b0646478cc8cb; no voters:
I20260924 04:07:03.135453 18473 raft_consensus.cc:2781] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:03.135667 18473 raft_consensus.cc:686] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 1 LEADER]: Becoming Leader. State: Replica: 98d058724c144509a2c795dfc10d9347, State: Running, Role: LEADER
I20260924 04:07:03.136078 18473 consensus_queue.cc:237] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 04:07:03.139324 17999 catalog_manager.cc:5495] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98d058724c144509a2c795dfc10d9347 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "98d058724c144509a2c795dfc10d9347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98d058724c144509a2c795dfc10d9347" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } health_report { overall_health: HEALTHY } } }
W20260924 04:07:03.146939 18450 tablet.cc:2369] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:07:03.172781 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:03.169177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.0.98.1:39255)
0924 04:07:03.169295 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:03.169299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:03.169312 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:07:03.169387 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:03.169389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:03.169446 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:03.169567 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:03.169784 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.170338 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.171289 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:03.171550 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:03.171633 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:03.171882 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:03.172360 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:03.172451 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:03.172574 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 04:07:03.172630 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:07:03.179576 18403 raft_consensus.cc:1261] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 1 FOLLOWER]: Refusing update from remote peer 98d058724c144509a2c795dfc10d9347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:03.179589 18272 raft_consensus.cc:1261] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 1 FOLLOWER]: Refusing update from remote peer 98d058724c144509a2c795dfc10d9347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:03.180361 18473 consensus_queue.cc:1035] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f426e657edad44ab856b0646478cc8cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:03.180598 18477 consensus_queue.cc:1035] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5dd3bb2e164bf8a061d802c451e5fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:03.189308 18481 mvcc.cc:205] Tried to move back new op lower bound from 7332752683735011328 to 7332752683568111616. Current Snapshot: MvccSnapshot[applied={T|T < 7332752683735011328}]
I20260924 04:07:03.197129 17998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:03.197304 17998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:03.199878 17998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 900e813229364a01af238436b74dd8ef: Sending DeleteTablet for 3 replicas of tablet 45f31820fe564adf997cdee867c138dc
I20260924 04:07:03.200842 18252 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f31820fe564adf997cdee867c138dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:03 UTC) from {username='slave'} at 127.0.0.1:35224
I20260924 04:07:03.201001 18121 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f31820fe564adf997cdee867c138dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:03 UTC) from {username='slave'} at 127.0.0.1:56674
I20260924 04:07:03.201426 18491 tablet_replica.cc:330] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347: stopping tablet replica
I20260924 04:07:03.201439 18490 tablet_replica.cc:330] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: stopping tablet replica
I20260924 04:07:03.201486 18383 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f31820fe564adf997cdee867c138dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:03 UTC) from {username='slave'} at 127.0.0.1:44642
I20260924 04:07:03.201782 18491 raft_consensus.cc:2229] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:07:03.201987 18490 raft_consensus.cc:2229] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:03.202169 18491 raft_consensus.cc:2258] T 45f31820fe564adf997cdee867c138dc P 98d058724c144509a2c795dfc10d9347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:03.202390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 18059
I20260924 04:07:03.203807 18490 raft_consensus.cc:2258] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:03.205672 18490 ts_tablet_manager.cc:1905] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:03.206037 18492 tablet_replica.cc:330] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: stopping tablet replica
I20260924 04:07:03.206316 18492 raft_consensus.cc:2229] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:03.206637 18492 raft_consensus.cc:2258] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:03.207937 18492 ts_tablet_manager.cc:1905] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:03.208947 18490 ts_tablet_manager.cc:1918] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:03.209044 18490 log.cc:1196] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/45f31820fe564adf997cdee867c138dc
I20260924 04:07:03.209383 18490 ts_tablet_manager.cc:1939] T 45f31820fe564adf997cdee867c138dc P f426e657edad44ab856b0646478cc8cb: Deleting consensus metadata
W20260924 04:07:03.209605 17986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39255 (error 108)
I20260924 04:07:03.209818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 18190
I20260924 04:07:03.210600 18492 ts_tablet_manager.cc:1918] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:03.210696 18492 log.cc:1196] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/45f31820fe564adf997cdee867c138dc
I20260924 04:07:03.210984 18492 ts_tablet_manager.cc:1939] T 45f31820fe564adf997cdee867c138dc P fc5dd3bb2e164bf8a061d802c451e5fa: Deleting consensus metadata
I20260924 04:07:03.211118 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:03.210800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39255 (local address 127.0.0.1:56684)
0924 04:07:03.210949 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:03.211027 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 04:07:03.211341 17986 catalog_manager.cc:4584] TS 98d058724c144509a2c795dfc10d9347 (127.0.98.1:39255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45f31820fe564adf997cdee867c138dc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39255: connect: Connection refused (error 111)
I20260924 04:07:03.212185 17983 catalog_manager.cc:4841] TS fc5dd3bb2e164bf8a061d802c451e5fa (127.0.98.3:45371): tablet 45f31820fe564adf997cdee867c138dc (table test-table [id=d9849f78aceb47f68c2283964c90bf39]) successfully deleted
I20260924 04:07:03.215862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 18321
I20260924 04:07:03.220264 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:03.216111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35001 (local address 127.0.0.1:35228)
0924 04:07:03.220115 (+ 4004us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:03.220179 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3901}
I20260924 04:07:03.221247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 17968
2026-09-24T04:07:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1167 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:03 dist-test-slave-zcsk krb5kdc[18496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:03 dist-test-slave-zcsk krb5kdc[18496](info): set up 2 sockets
Sep 24 04:07:03 dist-test-slave-zcsk krb5kdc[18496](info): commencing operation
krb5kdc: starting...
W20260924 04:07:04.175625 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:04 dist-test-slave-zcsk krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:04.293023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37871
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:04.402052 18509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:04.402328 18509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:04.402369 18509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:04.406399 18509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:04.406472 18509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:04.406491 18509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:04.406507 18509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:04.406522 18509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:04.412168 18509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:04.413323 18509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:04.414384 18509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:04.423056 18517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:04.423103 18515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:04.423055 18514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:04.423419 18509 server_base.cc:1034] running on GCE node
I20260924 04:07:04.424160 18509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:04.425602 18509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:04.426782 18509 hybrid_clock.cc:648] HybridClock initialized: now 1790222824426757 us; error 40 us; skew 500 ppm
Sep 24 04:07:04 dist-test-slave-zcsk krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:04.429551 18509 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:04.430459 18509 webserver.cc:460] Webserver started at http://127.0.98.62:39699/ using document root <none> and password file <none>
I20260924 04:07:04.431002 18509 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:04.431070 18509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:04.431280 18509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:04.433393 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "22351a669e4649f6af47f8f2cd9fef61"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "1920368e76f00d23902bf4207adec209"
server_key_iv: "bcd0940314162bc4493383650c7f71c3"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.433965 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "22351a669e4649f6af47f8f2cd9fef61"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "1920368e76f00d23902bf4207adec209"
server_key_iv: "bcd0940314162bc4493383650c7f71c3"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.437875 18509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:07:04.440367 18524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:04.441668 18509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:04.441777 18509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "22351a669e4649f6af47f8f2cd9fef61"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "1920368e76f00d23902bf4207adec209"
server_key_iv: "bcd0940314162bc4493383650c7f71c3"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.441870 18509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:04.466457 18509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:04.469349 18509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:04.469516 18509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:04.476531 18509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37871
I20260924 04:07:04.476564 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37871 every 8 connection(s)
I20260924 04:07:04.477545 18509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:04.479332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 18509
I20260924 04:07:04.479466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:04.479779 392 external_mini_cluster.cc:1428] Setting key 330a1ca45cda2709ba01de0a50f4e823
I20260924 04:07:04.480772 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:04.487160 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61: Bootstrap starting.
Sep 24 04:07:04 dist-test-slave-zcsk krb5kdc[18496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:04.489871 18577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:04.490579 18577 log.cc:824] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:04.492514 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61: No bootstrap required, opened a new log
I20260924 04:07:04.494084 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:04.481191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.0.98.62:37871)
0924 04:07:04.481599 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:04.481615 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:04.481650 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:07:04.482293 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:04.482308 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:04.482631 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:04.482919 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:04.483780 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.484838 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.486015 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.486496 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.486612 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:04.489378 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:04.489385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:04.489404 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:04.489436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:04.491083 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:04.491526 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:04.491531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:04.491536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:04.491649 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:04.491926 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:04.491929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:04.491931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:04.492334 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:04.492513 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:04.492821 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:07:04.493136 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":129,"threads_started":1}
I20260924 04:07:04.496449 18577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22351a669e4649f6af47f8f2cd9fef61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37871 } }
I20260924 04:07:04.496680 18577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:04.496721 18577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22351a669e4649f6af47f8f2cd9fef61, State: Initialized, Role: FOLLOWER
I20260924 04:07:04.497135 18577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22351a669e4649f6af47f8f2cd9fef61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37871 } }
I20260924 04:07:04.497268 18577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:04.497314 18577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:04.497388 18577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:04.498296 18577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22351a669e4649f6af47f8f2cd9fef61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37871 } }
I20260924 04:07:04.498623 18577 leader_election.cc:302] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22351a669e4649f6af47f8f2cd9fef61; no voters:
I20260924 04:07:04.498986 18577 leader_election.cc:288] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:04.499092 18582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:04.499346 18582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [term 1 LEADER]: Becoming Leader. State: Replica: 22351a669e4649f6af47f8f2cd9fef61, State: Running, Role: LEADER
I20260924 04:07:04.499686 18582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22351a669e4649f6af47f8f2cd9fef61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37871 } }
I20260924 04:07:04.500197 18577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:04.501374 18584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22351a669e4649f6af47f8f2cd9fef61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22351a669e4649f6af47f8f2cd9fef61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37871 } } }
I20260924 04:07:04.501406 18583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22351a669e4649f6af47f8f2cd9fef61. Latest consensus state: current_term: 1 leader_uuid: "22351a669e4649f6af47f8f2cd9fef61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22351a669e4649f6af47f8f2cd9fef61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37871 } } }
I20260924 04:07:04.501513 18583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:04.501511 18584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:04.501945 18592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:04.505151 18592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:04.510403 18592 catalog_manager.cc:1347] Generated new cluster ID: b7d1542eb5124389bf35cd84303a13cd
I20260924 04:07:04.510481 18592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:04.532395 18592 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:04.533433 18592 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:04.537168 18592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61: Generated new TSK 0
I20260924 04:07:04.537904 18592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:07:04.569212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37871
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:04.680162 18603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:04.680433 18603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:04.680475 18603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:04.684548 18603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:04.684613 18603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:04.684680 18603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:07:04.689980 18603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:04.691058 18603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:04.692116 18603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:04.701666 18609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:04.701711 18611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:04.701753 18608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:04.701855 18603 server_base.cc:1034] running on GCE node
I20260924 04:07:04.702389 18603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:04.703048 18603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:04.704234 18603 hybrid_clock.cc:648] HybridClock initialized: now 1790222824704214 us; error 33 us; skew 500 ppm
Sep 24 04:07:04 dist-test-slave-zcsk krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:04.706902 18603 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:07:04.707748 18603 webserver.cc:460] Webserver started at http://127.0.98.1:34207/ using document root <none> and password file <none>
I20260924 04:07:04.708313 18603 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:04.708374 18603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:04.708545 18603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:04.710539 18603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "01d70205af8549569aa823dbd23ceacf"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "eb9584303c6c3eb8deb49c7cf0be3c33"
server_key_iv: "31a3579f65bbb9c1eda4d39fab555cc4"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.711058 18603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01d70205af8549569aa823dbd23ceacf"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "eb9584303c6c3eb8deb49c7cf0be3c33"
server_key_iv: "31a3579f65bbb9c1eda4d39fab555cc4"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.714820 18603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:07:04.717463 18618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:04.718701 18603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:07:04.718814 18603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "01d70205af8549569aa823dbd23ceacf"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "eb9584303c6c3eb8deb49c7cf0be3c33"
server_key_iv: "31a3579f65bbb9c1eda4d39fab555cc4"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.718930 18603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:04.744817 18603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:04.748231 18603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:04.748421 18603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:04.749017 18603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:04.749899 18603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:04.749946 18603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:04.750001 18603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:04.750026 18603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:04.758499 18603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33171
I20260924 04:07:04.758571 18731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33171 every 8 connection(s)
I20260924 04:07:04.759424 18603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:07:04.765480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 18603
I20260924 04:07:04.765596 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:07:04.765883 392 external_mini_cluster.cc:1428] Setting key c1bfae1a16461492f49eb656da941619
Sep 24 04:07:04 dist-test-slave-zcsk krb5kdc[18496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:04.773947 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:04.761344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56417 (local address 127.0.98.62:37871)
0924 04:07:04.761540 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:04.761544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:04.762446 (+ 902us) server_negotiation.cc:407] Connection header received
0924 04:07:04.763408 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:04.763412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:04.763468 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:04.763542 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:04.764809 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.765354 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.766708 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.766948 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.767035 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:04.770022 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:04.770026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:04.770028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:04.770055 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:04.771581 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:04.772239 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:04.772243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:04.772245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:04.772337 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:04.772712 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:04.772715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:04.772716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:04.772927 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:04.773024 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:04.773600 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 04:07:04.773759 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:07:04.774768 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:04.761757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37871 (local address 127.0.98.1:56417)
0924 04:07:04.762281 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:04.762329 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:04.763185 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:04.763720 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:04.763727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:04.764145 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:04.764652 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:04.764667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.765480 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.765483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:04.766581 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:04.766592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.767091 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.767095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:04.767274 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:04.767899 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:04.767947 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:04.769811 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:04.771734 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:04.771742 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:04.771761 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:04.772123 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:04.772429 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:04.772433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:04.772435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:04.772591 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:04.773037 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:04.773045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:04.773335 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 04:07:04.773520 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 04:07:04.773886 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":118,"threads_started":1}
I20260924 04:07:04.776120 18732 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37871
I20260924 04:07:04.776382 18732 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:04.776932 18732 heartbeater.cc:511] Master 127.0.98.62:37871 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:07:04.778519 18541 ts_manager.cc:194] Registered new tserver with Master: 01d70205af8549569aa823dbd23ceacf (127.0.98.1:33171)
I20260924 04:07:04.780094 18541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:07:04.793428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37871
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:04.904147 18737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:04.904402 18737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:04.904444 18737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:04.908402 18737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:04.908459 18737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:04.908540 18737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:07:04.913878 18737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:04.914907 18737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:04.915871 18737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:04.925609 18743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:04.925693 18737 server_base.cc:1034] running on GCE node
W20260924 04:07:04.925609 18742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:04.925638 18745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:04.926244 18737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:04.926916 18737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:04.928074 18737 hybrid_clock.cc:648] HybridClock initialized: now 1790222824928041 us; error 47 us; skew 500 ppm
Sep 24 04:07:04 dist-test-slave-zcsk krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:04.936651 18737 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:07:04.937655 18737 webserver.cc:460] Webserver started at http://127.0.98.2:39491/ using document root <none> and password file <none>
I20260924 04:07:04.938215 18737 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:04.938275 18737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:04.938452 18737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:04.940649 18737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "c54a7297210543479650b77aed58908d"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "a6043976403e1ce0e6859df6edfee0a3"
server_key_iv: "0678e7604572539e84c1171d0752e8e4"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.941171 18737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c54a7297210543479650b77aed58908d"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "a6043976403e1ce0e6859df6edfee0a3"
server_key_iv: "0678e7604572539e84c1171d0752e8e4"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.944877 18737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:04.947327 18752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:04.948604 18737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:07:04.948707 18737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "c54a7297210543479650b77aed58908d"
format_stamp: "Formatted at 2026-09-24 04:07:04 on dist-test-slave-zcsk"
server_key: "a6043976403e1ce0e6859df6edfee0a3"
server_key_iv: "0678e7604572539e84c1171d0752e8e4"
server_key_version: "encryptionkey@0"
I20260924 04:07:04.948807 18737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:04.967330 18737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:04.970387 18737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:04.970546 18737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:04.971122 18737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:04.972064 18737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:04.972119 18737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:04.972160 18737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:04.972174 18737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:04.980847 18737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39789
I20260924 04:07:04.980851 18865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39789 every 8 connection(s)
I20260924 04:07:04.981738 18737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:07:04.990231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 18737
I20260924 04:07:04.990350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
Sep 24 04:07:04 dist-test-slave-zcsk krb5kdc[18496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:04.990617 392 external_mini_cluster.cc:1428] Setting key 8c2e135c6a1436caccafb7dcc7d4ca89
I20260924 04:07:04.995100 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:04.983552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44941 (local address 127.0.98.62:37871)
0924 04:07:04.983697 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:04.983702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:04.984621 (+ 919us) server_negotiation.cc:407] Connection header received
0924 04:07:04.985470 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:04.985473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:04.985527 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:04.985599 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:04.986746 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.987307 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.988495 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.988721 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.988808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:04.991214 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:04.991217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:04.991219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:04.991248 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:04.992874 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:04.993386 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:04.993390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:04.993392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:04.993473 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:04.993849 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:04.993853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:04.993855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:04.994086 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:04.994157 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:04.994869 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 04:07:04.994979 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:07:04.996119 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:04.983997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37871 (local address 127.0.98.2:44941)
0924 04:07:04.984456 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:04.984501 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:04.985276 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:04.985733 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:04.985740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:04.986098 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:04.986583 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:04.986598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.987418 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.987420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:04.988383 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:04.988392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:04.988855 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:04.988858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:04.989006 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:04.989510 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:04.989526 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:04.991048 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:04.993011 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:04.993016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:04.993030 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:04.993286 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:04.993553 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:04.993556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:04.993559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:04.993718 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:04.994236 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:04.994243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:04.994580 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 04:07:04.994799 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 04:07:04.995184 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260924 04:07:04.997349 18866 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37871
I20260924 04:07:04.997587 18866 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:04.998091 18866 heartbeater.cc:511] Master 127.0.98.62:37871 requested a full tablet report, sending...
I20260924 04:07:04.999136 18541 ts_manager.cc:194] Registered new tserver with Master: c54a7297210543479650b77aed58908d (127.0.98.2:39789)
I20260924 04:07:04.999971 18541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44941
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:07:05.018646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37871
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:05.129813 18871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:05.130103 18871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:05.130146 18871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:05.134173 18871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:05.134241 18871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:05.134301 18871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:07:05.139688 18871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:05.140836 18871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:05.141856 18871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:05.151690 18876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:05.151696 18877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:05.151862 18871 server_base.cc:1034] running on GCE node
W20260924 04:07:05.151706 18879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:05.152426 18871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:05.153120 18871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:05.154285 18871 hybrid_clock.cc:648] HybridClock initialized: now 1790222825154257 us; error 32 us; skew 500 ppm
Sep 24 04:07:05 dist-test-slave-zcsk krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:05.157054 18871 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:07:05.158018 18871 webserver.cc:460] Webserver started at http://127.0.98.3:42079/ using document root <none> and password file <none>
I20260924 04:07:05.158576 18871 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:05.158643 18871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:05.158856 18871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:05.160993 18871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c7bef849a20457a9df57b2af31fc98d"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "e33593ba0319760cda1893043ac27c23"
server_key_iv: "a576f7faf9e5f2869411e030c8a39693"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.161538 18871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c7bef849a20457a9df57b2af31fc98d"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "e33593ba0319760cda1893043ac27c23"
server_key_iv: "a576f7faf9e5f2869411e030c8a39693"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.165347 18871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:07:05.167866 18886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:05.169124 18871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:05.169234 18871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "5c7bef849a20457a9df57b2af31fc98d"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "e33593ba0319760cda1893043ac27c23"
server_key_iv: "a576f7faf9e5f2869411e030c8a39693"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.169353 18871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:05.183501 18871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:05.186412 18871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:05.186592 18871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:05.187193 18871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:05.188066 18871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:05.188114 18871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:05.188167 18871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:05.188201 18871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:05.196765 18871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36249
I20260924 04:07:05.196791 18999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36249 every 8 connection(s)
I20260924 04:07:05.197701 18871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:07:05.204898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 18871
I20260924 04:07:05.205040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:07:05.205338 392 external_mini_cluster.cc:1428] Setting key c91fb99029335c26f032b92e10e85609
Sep 24 04:07:05 dist-test-slave-zcsk krb5kdc[18496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:05.210753 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.199456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46105 (local address 127.0.98.62:37871)
0924 04:07:05.199624 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.199629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.200557 (+ 928us) server_negotiation.cc:407] Connection header received
0924 04:07:05.201522 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.201524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.201574 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.201647 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:05.202809 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.203334 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.204467 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.204763 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.204879 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.207534 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:05.207539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:05.207541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:05.207565 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:05.208813 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:05.209280 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:05.209283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:05.209284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:05.209366 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:05.209690 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:05.209693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:05.209695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:05.209887 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:05.209974 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.210474 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.210597 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:07:05.211587 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.199886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37871 (local address 127.0.98.3:46105)
0924 04:07:05.200412 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:05.200458 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:05.201324 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:05.201780 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:05.201786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:05.202139 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:05.202676 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.202690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.203430 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.203432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.204336 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.204345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.204907 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.204909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.205082 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.205668 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:05.205689 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:05.207385 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:05.208920 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:05.208926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:05.208944 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:05.209187 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:05.209455 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:05.209458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:05.209461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:05.209605 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:05.209988 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:05.209995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:05.210246 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 04:07:05.210418 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 04:07:05.210769 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
I20260924 04:07:05.212759 19000 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37871
I20260924 04:07:05.213016 19000 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:05.213519 19000 heartbeater.cc:511] Master 127.0.98.62:37871 requested a full tablet report, sending...
I20260924 04:07:05.214553 18541 ts_manager.cc:194] Registered new tserver with Master: 5c7bef849a20457a9df57b2af31fc98d (127.0.98.3:36249)
I20260924 04:07:05.215096 18541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46105
I20260924 04:07:05.219223 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:07:05.228384 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.221131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.0.98.62:37871)
0924 04:07:05.221264 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.221269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.221406 (+ 137us) server_negotiation.cc:407] Connection header received
0924 04:07:05.221501 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.221503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.221542 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.221603 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:05.222236 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.222791 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.223705 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.223948 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.224035 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.225469 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:05.225471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:05.225473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:05.225494 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:05.226624 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:05.227114 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:05.227117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:05.227118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:05.227251 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:05.227530 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:05.227533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:05.227534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:05.227752 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:05.227869 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.228106 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.228191 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:07:05.231144 18541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:07:05.234071 18541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:07:05.248690 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.242516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36249 (local address 127.0.0.1:34752)
0924 04:07:05.243270 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:05.243284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:05.243370 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:05.244398 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:05.244400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:05.244417 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:05.244500 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.244506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.245662 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.245666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.247286 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.247292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.248130 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.248133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.248239 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.248274 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:07:05.248382 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:07:05.248507 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":135,"thread_start_us":111,"threads_started":1}
I20260924 04:07:05.248680 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.242846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.0.98.3:36249)
0924 04:07:05.243556 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.243567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.243591 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:07:05.243668 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.243674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.243887 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.244098 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:05.244589 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.245545 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.247409 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.248020 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.248116 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.248278 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.248411 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.248506 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":142,"threads_started":1}
I20260924 04:07:05.248741 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.242403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33171 (local address 127.0.0.1:45834)
0924 04:07:05.242730 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:05.242776 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:05.242897 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:05.244106 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:05.244111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:05.244133 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:05.244271 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.244278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.245709 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.245711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.247261 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.247273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.248326 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.248330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.248418 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.248440 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:07:05.248491 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:07:05.248565 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":111,"thread_start_us":73,"threads_started":1}
I20260924 04:07:05.249059 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.243339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.0.98.2:39789)
0924 04:07:05.243701 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.243710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.244451 (+ 741us) server_negotiation.cc:407] Connection header received
0924 04:07:05.244553 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.244558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.244739 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.244882 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:05.245057 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.246003 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.247314 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.248188 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.248296 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.248439 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.248808 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.248906 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":106,"threads_started":1}
I20260924 04:07:05.249300 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.243114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45834 (local address 127.0.98.1:33171)
0924 04:07:05.243621 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.243638 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.243659 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:07:05.243734 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.243739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.243970 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.244126 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:05.244387 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.245593 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.247393 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.248188 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.248856 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.248998 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.249045 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.249151 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":120,"threads_started":1}
I20260924 04:07:05.249554 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.243125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39789 (local address 127.0.0.1:32818)
0924 04:07:05.244352 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:05.244364 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:05.244455 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:05.244874 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:05.244877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:05.244895 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:05.244967 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.244973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.246120 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.246123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.247202 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.247211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.248495 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.248499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.248573 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.248592 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:07:05.248664 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:07:05.248723 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":27,"spinlock_wait_cycles":9344,"thread_start_us":85,"threads_started":1}
I20260924 04:07:05.252137 18934 tablet_service.cc:1467] Processing CreateTablet for tablet 0117ef60acb2462ab097a855698bd9d0 (DEFAULT_TABLE table=test-table [id=7e4c3af7411d45febb7cc3e4dc3039c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:05.252345 18666 tablet_service.cc:1467] Processing CreateTablet for tablet 0117ef60acb2462ab097a855698bd9d0 (DEFAULT_TABLE table=test-table [id=7e4c3af7411d45febb7cc3e4dc3039c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:05.252741 18800 tablet_service.cc:1467] Processing CreateTablet for tablet 0117ef60acb2462ab097a855698bd9d0 (DEFAULT_TABLE table=test-table [id=7e4c3af7411d45febb7cc3e4dc3039c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:05.253260 18934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0117ef60acb2462ab097a855698bd9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:05.253430 18666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0117ef60acb2462ab097a855698bd9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:05.253655 18800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0117ef60acb2462ab097a855698bd9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:05.260263 19017 tablet_bootstrap.cc:492] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: Bootstrap starting.
I20260924 04:07:05.261283 19019 tablet_bootstrap.cc:492] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: Bootstrap starting.
I20260924 04:07:05.261397 19018 tablet_bootstrap.cc:492] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf: Bootstrap starting.
I20260924 04:07:05.262523 19017 tablet_bootstrap.cc:654] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:05.263511 19019 tablet_bootstrap.cc:654] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:05.263511 19018 tablet_bootstrap.cc:654] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:05.263558 19017 log.cc:824] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:05.264792 19019 log.cc:824] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:05.264792 19018 log.cc:824] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:05.265697 19017 tablet_bootstrap.cc:492] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: No bootstrap required, opened a new log
I20260924 04:07:05.266001 19017 ts_tablet_manager.cc:1397] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260924 04:07:05.267043 19018 tablet_bootstrap.cc:492] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf: No bootstrap required, opened a new log
I20260924 04:07:05.267050 19019 tablet_bootstrap.cc:492] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: No bootstrap required, opened a new log
I20260924 04:07:05.267243 19018 ts_tablet_manager.cc:1397] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:07:05.267253 19019 ts_tablet_manager.cc:1397] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:07:05.270074 19018 raft_consensus.cc:348] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.270097 19017 raft_consensus.cc:348] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.270320 19018 raft_consensus.cc:374] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:05.270368 19018 raft_consensus.cc:729] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d70205af8549569aa823dbd23ceacf, State: Initialized, Role: FOLLOWER
I20260924 04:07:05.270372 19017 raft_consensus.cc:374] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:05.270421 19017 raft_consensus.cc:729] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7bef849a20457a9df57b2af31fc98d, State: Initialized, Role: FOLLOWER
I20260924 04:07:05.270854 19018 consensus_queue.cc:260] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.270901 19017 consensus_queue.cc:260] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.271090 19019 raft_consensus.cc:348] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.271478 19019 raft_consensus.cc:374] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:05.271531 18732 heartbeater.cc:503] Master 127.0.98.62:37871 was elected leader, sending a full tablet report...
I20260924 04:07:05.271579 19019 raft_consensus.cc:729] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54a7297210543479650b77aed58908d, State: Initialized, Role: FOLLOWER
I20260924 04:07:05.271938 19018 ts_tablet_manager.cc:1428] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 04:07:05.272015 19017 ts_tablet_manager.cc:1428] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260924 04:07:05.272017 19019 consensus_queue.cc:260] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.272358 19000 heartbeater.cc:503] Master 127.0.98.62:37871 was elected leader, sending a full tablet report...
I20260924 04:07:05.272964 19019 ts_tablet_manager.cc:1428] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:07:05.273276 18866 heartbeater.cc:503] Master 127.0.98.62:37871 was elected leader, sending a full tablet report...
I20260924 04:07:05.320964 19023 raft_consensus.cc:482] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:07:05.321185 19023 raft_consensus.cc:504] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.322481 19023 leader_election.cc:288] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c7bef849a20457a9df57b2af31fc98d (127.0.98.3:36249), c54a7297210543479650b77aed58908d (127.0.98.2:39789)
I20260924 04:07:05.327250 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.322811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52581 (local address 127.0.98.3:36249)
0924 04:07:05.322933 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.322936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.323079 (+ 143us) server_negotiation.cc:407] Connection header received
0924 04:07:05.323215 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.323218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.323273 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.323359 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:05.323633 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.324427 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.326005 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.326623 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.326731 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.326804 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.327029 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.327100 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:07:05.327234 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.323227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43849 (local address 127.0.98.2:39789)
0924 04:07:05.323378 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.323383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.323495 (+ 112us) server_negotiation.cc:407] Connection header received
0924 04:07:05.323578 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.323580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.323624 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.323699 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:05.323936 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.324625 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.326002 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.326640 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.326731 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.326795 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.327030 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.327100 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:07:05.327279 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.322723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36249 (local address 127.0.98.1:52581)
0924 04:07:05.322998 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:05.323012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:05.323116 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:05.323382 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:05.323385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:05.323404 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:05.323538 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.323545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.324533 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.324535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.325883 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.325895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.326742 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.326746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.326886 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.326933 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 04:07:05.327010 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:07:05.327102 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":61,"thread_start_us":100,"threads_started":1}
I20260924 04:07:05.327283 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.323116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39789 (local address 127.0.98.1:43849)
0924 04:07:05.323410 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:05.323422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:05.323506 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:05.323712 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:05.323714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:05.323727 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:05.323819 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.323824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.324718 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.324721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.325883 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.325895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.326742 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.326746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.326913 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.326945 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 04:07:05.327010 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:07:05.327102 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":98,"thread_start_us":108,"threads_started":1}
I20260924 04:07:05.328018 18954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0117ef60acb2462ab097a855698bd9d0" candidate_uuid: "01d70205af8549569aa823dbd23ceacf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7bef849a20457a9df57b2af31fc98d" is_pre_election: true
I20260924 04:07:05.328018 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0117ef60acb2462ab097a855698bd9d0" candidate_uuid: "01d70205af8549569aa823dbd23ceacf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54a7297210543479650b77aed58908d" is_pre_election: true
I20260924 04:07:05.328358 18954 raft_consensus.cc:2454] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d70205af8549569aa823dbd23ceacf in term 0.
I20260924 04:07:05.328358 18820 raft_consensus.cc:2454] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d70205af8549569aa823dbd23ceacf in term 0.
I20260924 04:07:05.328850 18620 leader_election.cc:302] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d70205af8549569aa823dbd23ceacf, 5c7bef849a20457a9df57b2af31fc98d; no voters:
I20260924 04:07:05.329105 19023 raft_consensus.cc:2781] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:07:05.329190 19023 raft_consensus.cc:482] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:07:05.329217 19023 raft_consensus.cc:3037] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:05.330238 19023 raft_consensus.cc:504] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.330564 19023 leader_election.cc:288] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [CANDIDATE]: Term 1 election: Requested vote from peers 5c7bef849a20457a9df57b2af31fc98d (127.0.98.3:36249), c54a7297210543479650b77aed58908d (127.0.98.2:39789)
I20260924 04:07:05.330929 18954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0117ef60acb2462ab097a855698bd9d0" candidate_uuid: "01d70205af8549569aa823dbd23ceacf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7bef849a20457a9df57b2af31fc98d"
I20260924 04:07:05.330948 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0117ef60acb2462ab097a855698bd9d0" candidate_uuid: "01d70205af8549569aa823dbd23ceacf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c54a7297210543479650b77aed58908d"
I20260924 04:07:05.331058 18954 raft_consensus.cc:3037] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:05.331058 18820 raft_consensus.cc:3037] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:05.332365 18820 raft_consensus.cc:2454] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d70205af8549569aa823dbd23ceacf in term 1.
I20260924 04:07:05.332365 18954 raft_consensus.cc:2454] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d70205af8549569aa823dbd23ceacf in term 1.
I20260924 04:07:05.332675 18620 leader_election.cc:302] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d70205af8549569aa823dbd23ceacf, 5c7bef849a20457a9df57b2af31fc98d; no voters:
I20260924 04:07:05.332870 19023 raft_consensus.cc:2781] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:05.333101 19023 raft_consensus.cc:686] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [term 1 LEADER]: Becoming Leader. State: Replica: 01d70205af8549569aa823dbd23ceacf, State: Running, Role: LEADER
I20260924 04:07:05.333457 19023 consensus_queue.cc:237] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } }
I20260924 04:07:05.336673 18540 catalog_manager.cc:5495] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d70205af8549569aa823dbd23ceacf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "01d70205af8549569aa823dbd23ceacf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d70205af8549569aa823dbd23ceacf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 } health_report { overall_health: UNKNOWN } } }
I20260924 04:07:05.348086 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.344632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.0.98.1:33171)
0924 04:07:05.344774 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.344779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.344796 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:07:05.344896 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.344898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.344952 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.345037 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:05.345266 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.345806 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.346638 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.346855 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.346941 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.347240 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:05.347661 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:05.347803 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.347894 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.347952 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 04:07:05.354380 18954 raft_consensus.cc:1261] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 1 FOLLOWER]: Refusing update from remote peer 01d70205af8549569aa823dbd23ceacf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:05.354380 18820 raft_consensus.cc:1261] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 1 FOLLOWER]: Refusing update from remote peer 01d70205af8549569aa823dbd23ceacf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:05.355173 19028 consensus_queue.cc:1035] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54a7297210543479650b77aed58908d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:05.355406 19023 consensus_queue.cc:1035] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7bef849a20457a9df57b2af31fc98d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:05.362887 19031 mvcc.cc:205] Tried to move back new op lower bound from 7332752692643643392 to 7332752692568506368. Current Snapshot: MvccSnapshot[applied={T|T < 7332752692643643392}]
I20260924 04:07:05.364051 19033 mvcc.cc:205] Tried to move back new op lower bound from 7332752692643643392 to 7332752692568506368. Current Snapshot: MvccSnapshot[applied={T|T < 7332752692643643392}]
I20260924 04:07:05.372967 18540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:05.373139 18540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:05.375641 18540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22351a669e4649f6af47f8f2cd9fef61: Sending DeleteTablet for 3 replicas of tablet 0117ef60acb2462ab097a855698bd9d0
I20260924 04:07:05.376562 18666 tablet_service.cc:1514] Processing DeleteTablet for tablet 0117ef60acb2462ab097a855698bd9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45834
I20260924 04:07:05.376564 18800 tablet_service.cc:1514] Processing DeleteTablet for tablet 0117ef60acb2462ab097a855698bd9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32818
I20260924 04:07:05.377126 19041 tablet_replica.cc:330] T 0117ef60acb2462ab097a855698bd9d0 P 01d70205af8549569aa823dbd23ceacf: stopping tablet replica
I20260924 04:07:05.377276 18934 tablet_service.cc:1514] Processing DeleteTablet for tablet 0117ef60acb2462ab097a855698bd9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34752
I20260924 04:07:05.377403 19042 tablet_replica.cc:330] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: stopping tablet replica
I20260924 04:07:05.377669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 18603
I20260924 04:07:05.377763 19042 raft_consensus.cc:2229] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:05.377853 19043 tablet_replica.cc:330] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: stopping tablet replica
I20260924 04:07:05.378214 19042 raft_consensus.cc:2258] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:05.378255 19043 raft_consensus.cc:2229] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:05.378553 19043 raft_consensus.cc:2258] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:05.379963 19042 ts_tablet_manager.cc:1905] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:05.382157 19042 ts_tablet_manager.cc:1918] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:05.382259 19042 log.cc:1196] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/0117ef60acb2462ab097a855698bd9d0
I20260924 04:07:05.382514 19042 ts_tablet_manager.cc:1939] T 0117ef60acb2462ab097a855698bd9d0 P c54a7297210543479650b77aed58908d: Deleting consensus metadata
I20260924 04:07:05.383709 18526 catalog_manager.cc:4841] TS c54a7297210543479650b77aed58908d (127.0.98.2:39789): tablet 0117ef60acb2462ab097a855698bd9d0 (table test-table [id=7e4c3af7411d45febb7cc3e4dc3039c3]) successfully deleted
W20260924 04:07:05.384186 18528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33171 (error 108)
I20260924 04:07:05.384361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 18737
I20260924 04:07:05.385284 19043 ts_tablet_manager.cc:1905] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:05.386101 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.385795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33171 (local address 127.0.0.1:45854)
0924 04:07:05.385935 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:05.386010 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 04:07:05.386324 18528 catalog_manager.cc:4584] TS 01d70205af8549569aa823dbd23ceacf (127.0.98.1:33171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0117ef60acb2462ab097a855698bd9d0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33171: connect: Connection refused (error 111)
I20260924 04:07:05.387569 19043 ts_tablet_manager.cc:1918] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:05.387650 19043 log.cc:1196] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/0117ef60acb2462ab097a855698bd9d0
I20260924 04:07:05.387935 19043 ts_tablet_manager.cc:1939] T 0117ef60acb2462ab097a855698bd9d0 P 5c7bef849a20457a9df57b2af31fc98d: Deleting consensus metadata
I20260924 04:07:05.389361 18526 catalog_manager.cc:4841] TS 5c7bef849a20457a9df57b2af31fc98d (127.0.98.3:36249): tablet 0117ef60acb2462ab097a855698bd9d0 (table test-table [id=7e4c3af7411d45febb7cc3e4dc3039c3]) successfully deleted
I20260924 04:07:05.390015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 18871
I20260924 04:07:05.395406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 18509
2026-09-24T04:07:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2176 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T04:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:05Z Disabled control of system clock
I20260924 04:07:05.425987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36009
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:07:05.534284 19048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:05.534554 19048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:05.534590 19048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:05.538586 19048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:05.538651 19048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:05.538666 19048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:05.538681 19048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:05.538697 19048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:05.543668 19048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:05.544811 19048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:05.546238 19048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:05.554679 19053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:05.554679 19056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:05.554730 19048 server_base.cc:1034] running on GCE node
W20260924 04:07:05.554683 19054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:05.555536 19048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:05.556784 19048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:05.557963 19048 hybrid_clock.cc:648] HybridClock initialized: now 1790222825557916 us; error 62 us; skew 500 ppm
I20260924 04:07:05.559870 19048 webserver.cc:460] Webserver started at http://127.0.98.62:43151/ using document root <none> and password file <none>
I20260924 04:07:05.560479 19048 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:05.560544 19048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:05.560730 19048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:05.562834 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "9d68501285cc49ce9610d89d23867a35"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "e6ab66671017447367bb5853c6344b6b"
server_key_iv: "2f4bf01390b17e04278d0ed838b8d90d"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.563350 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d68501285cc49ce9610d89d23867a35"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "e6ab66671017447367bb5853c6344b6b"
server_key_iv: "2f4bf01390b17e04278d0ed838b8d90d"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.567040 19048 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 04:07:05.569476 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:05.570694 19048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:07:05.570825 19048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "9d68501285cc49ce9610d89d23867a35"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "e6ab66671017447367bb5853c6344b6b"
server_key_iv: "2f4bf01390b17e04278d0ed838b8d90d"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.570935 19048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:05.588835 19048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:05.589509 19048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:05.589659 19048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:05.596606 19048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36009
I20260924 04:07:05.596617 19114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36009 every 8 connection(s)
I20260924 04:07:05.597584 19048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:05.600697 19115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:05.601459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 19048
I20260924 04:07:05.601555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:05.601824 392 external_mini_cluster.cc:1428] Setting key cc814c4d3a3d6e594d917279ec1e6141
I20260924 04:07:05.606544 19115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35: Bootstrap starting.
I20260924 04:07:05.609179 19115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:05.609920 19115 log.cc:824] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:05.611159 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.603209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.98.62:36009)
0924 04:07:05.603841 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.603855 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.603890 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:07:05.604554 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.604570 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.604976 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.605279 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:05.605698 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.606649 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.607841 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.608331 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.608444 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.609274 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:05.609281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:05.609298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:05.609703 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:05.609769 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:05.609791 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:05.609862 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.610031 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.610327 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":127,"threads_started":1}
I20260924 04:07:05.611943 19115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35: No bootstrap required, opened a new log
I20260924 04:07:05.615278 19115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d68501285cc49ce9610d89d23867a35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36009 } }
I20260924 04:07:05.615495 19115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:05.615540 19115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d68501285cc49ce9610d89d23867a35, State: Initialized, Role: FOLLOWER
I20260924 04:07:05.615972 19115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d68501285cc49ce9610d89d23867a35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36009 } }
I20260924 04:07:05.616107 19115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:05.616176 19115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:05.616272 19115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:05.617170 19115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d68501285cc49ce9610d89d23867a35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36009 } }
I20260924 04:07:05.617487 19115 leader_election.cc:302] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d68501285cc49ce9610d89d23867a35; no voters:
I20260924 04:07:05.617779 19115 leader_election.cc:288] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:05.617882 19120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:05.618203 19120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [term 1 LEADER]: Becoming Leader. State: Replica: 9d68501285cc49ce9610d89d23867a35, State: Running, Role: LEADER
I20260924 04:07:05.618516 19120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d68501285cc49ce9610d89d23867a35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36009 } }
I20260924 04:07:05.619014 19115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:05.620134 19122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d68501285cc49ce9610d89d23867a35. Latest consensus state: current_term: 1 leader_uuid: "9d68501285cc49ce9610d89d23867a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d68501285cc49ce9610d89d23867a35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36009 } } }
I20260924 04:07:05.620262 19122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:05.620507 19121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d68501285cc49ce9610d89d23867a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d68501285cc49ce9610d89d23867a35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36009 } } }
I20260924 04:07:05.620601 19121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:05.620698 19129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:05.623440 19129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:05.628454 19129 catalog_manager.cc:1347] Generated new cluster ID: cf6579f711374d5a8f05248e49e775dc
I20260924 04:07:05.628535 19129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:05.646944 19129 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:05.648298 19129 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:05.657622 19129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35: Generated new TSK 0
I20260924 04:07:05.658509 19129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:05.667361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36009
--builtin_ntp_servers=127.0.98.20:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:07:05.784368 19139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:05.784826 19139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:05.784897 19139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:05.790261 19139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:05.790330 19139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:05.790390 19139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:07:05.795382 19139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:05.796550 19139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:05.797914 19139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:05.807647 19144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:05.807713 19145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:05.807667 19147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:05.807705 19139 server_base.cc:1034] running on GCE node
I20260924 04:07:05.808243 19139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:05.808920 19139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:05.810096 19139 hybrid_clock.cc:648] HybridClock initialized: now 1790222825810067 us; error 38 us; skew 500 ppm
I20260924 04:07:05.812071 19139 webserver.cc:460] Webserver started at http://127.0.98.1:35487/ using document root <none> and password file <none>
I20260924 04:07:05.812655 19139 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:05.812726 19139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:05.812939 19139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:05.815147 19139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "7ca3bd35ae2c433c9fa436db407cf635"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "a1695e89eff7f90b20be37e32c56167f"
server_key_iv: "514b2f08f8f704c910a295a1c66768a3"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.815829 19139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7ca3bd35ae2c433c9fa436db407cf635"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "a1695e89eff7f90b20be37e32c56167f"
server_key_iv: "514b2f08f8f704c910a295a1c66768a3"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.819684 19139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:07:05.822177 19153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:05.823390 19139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:07:05.823494 19139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "7ca3bd35ae2c433c9fa436db407cf635"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "a1695e89eff7f90b20be37e32c56167f"
server_key_iv: "514b2f08f8f704c910a295a1c66768a3"
server_key_version: "encryptionkey@0"
I20260924 04:07:05.823598 19139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:05.851336 19139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:05.852169 19139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:05.852358 19139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:05.852989 19139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:05.854022 19139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:05.854072 19139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:05.854130 19139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:05.854163 19139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:05.862795 19139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46525
I20260924 04:07:05.862825 19266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46525 every 8 connection(s)
I20260924 04:07:05.863757 19139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:07:05.864163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 19139
I20260924 04:07:05.864290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:07:05.864596 392 external_mini_cluster.cc:1428] Setting key 8b4374a3c5ddd3210a941dc9067c3c55
I20260924 04:07:05.866916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36009
--builtin_ntp_servers=127.0.98.20:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:07:05.874851 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.865780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46419 (local address 127.0.98.62:36009)
0924 04:07:05.865900 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:05.865905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:05.867385 (+ 1480us) server_negotiation.cc:407] Connection header received
0924 04:07:05.868864 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:05.868869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:05.868930 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:05.869034 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:05.870146 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.870994 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.872465 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.872817 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.872935 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.874314 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:05.874318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:05.874321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:05.874339 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:05.874366 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:05.874377 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:05.874455 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:05.874603 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 04:07:05.874682 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:07:05.875813 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:05.866430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36009 (local address 127.0.98.1:46419)
0924 04:07:05.867213 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:05.867269 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:05.868611 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:05.869306 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:05.869315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:05.869416 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:05.869990 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.870005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.871201 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.871204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.872342 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:05.872353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:05.873008 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:05.873011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:05.873224 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:05.873995 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:05.873997 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:05.874045 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:05.874128 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:05.874465 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:05.874473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:05.874477 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:07:05.874589 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 04:07:05.874845 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260924 04:07:05.877403 19267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36009
I20260924 04:07:05.877679 19267 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:05.878276 19267 heartbeater.cc:511] Master 127.0.98.62:36009 requested a full tablet report, sending...
I20260924 04:07:05.879863 19079 ts_manager.cc:194] Registered new tserver with Master: 7ca3bd35ae2c433c9fa436db407cf635 (127.0.98.1:46525)
I20260924 04:07:05.881150 19079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46419
W20260924 04:07:05.976511 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:05.976832 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:05.976892 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:05.980960 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:05.981025 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:05.981104 19270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:07:05.985989 19270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:05.987089 19270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:05.988516 19270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:05.997982 19275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:05.998003 19278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:05.997997 19276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:05.998245 19270 server_base.cc:1034] running on GCE node
I20260924 04:07:05.998654 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:05.999316 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:06.000466 19270 hybrid_clock.cc:648] HybridClock initialized: now 1790222826000444 us; error 33 us; skew 500 ppm
I20260924 04:07:06.002270 19270 webserver.cc:460] Webserver started at http://127.0.98.2:39175/ using document root <none> and password file <none>
I20260924 04:07:06.002831 19270 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:06.002897 19270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:06.003104 19270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:06.005192 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "18ed304ee72c4761927e5c21d23933ed"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "8eabdb6602b909fd5634630f662e2bde"
server_key_iv: "79808e2858b453de09f6ff823a708148"
server_key_version: "encryptionkey@0"
I20260924 04:07:06.005751 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "18ed304ee72c4761927e5c21d23933ed"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "8eabdb6602b909fd5634630f662e2bde"
server_key_iv: "79808e2858b453de09f6ff823a708148"
server_key_version: "encryptionkey@0"
I20260924 04:07:06.009553 19270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:07:06.012131 19284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:06.013497 19270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:07:06.013614 19270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "18ed304ee72c4761927e5c21d23933ed"
format_stamp: "Formatted at 2026-09-24 04:07:05 on dist-test-slave-zcsk"
server_key: "8eabdb6602b909fd5634630f662e2bde"
server_key_iv: "79808e2858b453de09f6ff823a708148"
server_key_version: "encryptionkey@0"
I20260924 04:07:06.013749 19270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:06.029559 19270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:06.030278 19270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:06.030462 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:06.031052 19270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:06.032037 19270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:06.032096 19270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:06.032159 19270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:06.032183 19270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:06.040410 19270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42207
I20260924 04:07:06.040460 19397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42207 every 8 connection(s)
I20260924 04:07:06.041326 19270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:07:06.043200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 19270
I20260924 04:07:06.043305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:07:06.043609 392 external_mini_cluster.cc:1428] Setting key a481f14c289323d77c1e49254c0401f4
I20260924 04:07:06.045648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36009
--builtin_ntp_servers=127.0.98.20:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:07:06.052755 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.043372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45613 (local address 127.0.98.62:36009)
0924 04:07:06.043537 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.043543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.045282 (+ 1739us) server_negotiation.cc:407] Connection header received
0924 04:07:06.046701 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.046705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.046787 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.046872 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:06.048112 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.048643 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.050434 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.050654 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.050749 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.052220 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:06.052224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:06.052227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:06.052241 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:06.052262 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:06.052270 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:06.052333 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.052552 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.052622 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:07:06.054011 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.043714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36009 (local address 127.0.98.2:45613)
0924 04:07:06.045122 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:06.045187 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:06.046474 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:06.047071 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:06.047081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:06.047210 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:06.047960 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.047981 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.048776 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.048779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.050292 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.050306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.050833 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.050837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.051065 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.051823 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:06.051825 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:06.051895 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:06.052027 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:06.052354 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:06.052363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:06.052369 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:07:06.052506 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 04:07:06.052863 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":131,"threads_started":1}
I20260924 04:07:06.055291 19398 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36009
I20260924 04:07:06.055533 19398 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:06.056120 19398 heartbeater.cc:511] Master 127.0.98.62:36009 requested a full tablet report, sending...
I20260924 04:07:06.057184 19079 ts_manager.cc:194] Registered new tserver with Master: 18ed304ee72c4761927e5c21d23933ed (127.0.98.2:42207)
I20260924 04:07:06.057940 19079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45613
W20260924 04:07:06.160766 19401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:06.161053 19401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:06.161093 19401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:06.165717 19401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:06.165797 19401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:06.165858 19401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:07:06.170853 19401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:06.172030 19401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:06.173452 19401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:06.183746 19409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:06.183746 19406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:06.183786 19407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:06.184139 19401 server_base.cc:1034] running on GCE node
I20260924 04:07:06.184610 19401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:06.185350 19401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:06.186528 19401 hybrid_clock.cc:648] HybridClock initialized: now 1790222826186491 us; error 51 us; skew 500 ppm
I20260924 04:07:06.188674 19401 webserver.cc:460] Webserver started at http://127.0.98.3:42729/ using document root <none> and password file <none>
I20260924 04:07:06.189306 19401 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:06.189378 19401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:06.189582 19401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:06.191742 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "a38444bfdc784ceba94d0285d100327c"
format_stamp: "Formatted at 2026-09-24 04:07:06 on dist-test-slave-zcsk"
server_key: "bba39665e3251059e10a842cb6965e47"
server_key_iv: "b41e6fc838a4d4bb26f38ad5f577718d"
server_key_version: "encryptionkey@0"
I20260924 04:07:06.192363 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a38444bfdc784ceba94d0285d100327c"
format_stamp: "Formatted at 2026-09-24 04:07:06 on dist-test-slave-zcsk"
server_key: "bba39665e3251059e10a842cb6965e47"
server_key_iv: "b41e6fc838a4d4bb26f38ad5f577718d"
server_key_version: "encryptionkey@0"
I20260924 04:07:06.196358 19401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:07:06.199003 19415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:06.200484 19401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:06.200609 19401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "a38444bfdc784ceba94d0285d100327c"
format_stamp: "Formatted at 2026-09-24 04:07:06 on dist-test-slave-zcsk"
server_key: "bba39665e3251059e10a842cb6965e47"
server_key_iv: "b41e6fc838a4d4bb26f38ad5f577718d"
server_key_version: "encryptionkey@0"
I20260924 04:07:06.200726 19401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:06.219120 19401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:06.219898 19401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:06.220108 19401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:06.220743 19401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:06.221695 19401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:06.221743 19401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:06.221798 19401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:06.221827 19401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:06.230449 19401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37327
I20260924 04:07:06.230509 19528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37327 every 8 connection(s)
I20260924 04:07:06.231437 19401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:07:06.232270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 19401
I20260924 04:07:06.232380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:07:06.232640 392 external_mini_cluster.cc:1428] Setting key 9189bc4fc90f3a73cb20ae069cbc746d
I20260924 04:07:06.241137 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.233410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39385 (local address 127.0.98.62:36009)
0924 04:07:06.233531 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.233536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.234936 (+ 1400us) server_negotiation.cc:407] Connection header received
0924 04:07:06.236368 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.236371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.236417 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.236482 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:06.237432 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.237984 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.239186 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.239431 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.239517 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.240641 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:06.240643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:06.240645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:06.240659 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:06.240681 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:06.240688 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:06.240746 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.240885 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.240950 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:07:06.242040 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.234010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36009 (local address 127.0.98.3:39385)
0924 04:07:06.234779 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:06.234833 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:06.236153 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:06.236634 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:06.236641 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:06.236754 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:06.237272 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.237285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.238131 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.238134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.239061 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.239070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.239566 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.239569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.239740 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.240365 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:06.240367 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:06.240411 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:06.240484 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:06.240754 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:06.240759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:06.240763 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:07:06.240860 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 04:07:06.241111 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
I20260924 04:07:06.243285 19529 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36009
I20260924 04:07:06.243549 19529 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:06.244154 19529 heartbeater.cc:511] Master 127.0.98.62:36009 requested a full tablet report, sending...
I20260924 04:07:06.245344 19079 ts_manager.cc:194] Registered new tserver with Master: a38444bfdc784ceba94d0285d100327c (127.0.98.3:37327)
I20260924 04:07:06.245954 19079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39385
I20260924 04:07:06.246629 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:07:06.253249 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.249255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.0.98.62:36009)
0924 04:07:06.249417 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.249421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.249459 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:07:06.249581 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.249584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.249642 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.249709 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:06.249990 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.250544 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.251383 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.251612 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.251714 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.252569 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:06.252572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:06.252574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:06.252588 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:06.252608 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:06.252616 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:06.252739 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.252996 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.253045 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 04:07:06.256850 19078 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:07:06.260200 19078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:07:06.275699 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.270292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.0.98.2:42207)
0924 04:07:06.270706 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.270719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.271083 (+ 364us) server_negotiation.cc:407] Connection header received
0924 04:07:06.271166 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.271171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.271366 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.271513 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:06.271886 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.273021 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.274313 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.275038 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.275144 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.275355 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.275414 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.275524 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":88,"threads_started":1}
I20260924 04:07:06.275689 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.269978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42207 (local address 127.0.0.1:55152)
0924 04:07:06.270862 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:06.270875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:06.270981 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:06.271671 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:06.271674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:06.271690 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:06.271774 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.271780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.273124 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.273127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.274157 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.274166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.275138 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.275142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.275245 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.275293 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 04:07:06.275388 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:07:06.275527 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
I20260924 04:07:06.276250 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.270015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.0.98.3:37327)
0924 04:07:06.270806 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.270817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.270848 (+ 31us) server_negotiation.cc:407] Connection header received
0924 04:07:06.270927 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.270932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.271159 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.271314 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:06.271680 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.272848 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.274403 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.275226 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.275321 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.275512 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.275962 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.276070 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":148,"threads_started":1}
I20260924 04:07:06.276384 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.270613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46525 (local address 127.0.0.1:57384)
0924 04:07:06.271677 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:06.271689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:06.271759 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:06.272384 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:06.272387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:06.272404 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:06.272509 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.272516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.273791 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.273794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.274909 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.274919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.276001 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.276004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.276093 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.276114 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:07:06.276179 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:07:06.276256 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"thread_start_us":77,"threads_started":1}
I20260924 04:07:06.276482 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.271020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.0.98.1:46525)
0924 04:07:06.272028 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.272044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.272063 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:07:06.272120 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.272124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.272258 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.272382 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:06.272623 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.273641 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.275040 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.275871 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.275999 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.276184 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.276232 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.276320 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":111,"threads_started":1}
I20260924 04:07:06.276535 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.269883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37327 (local address 127.0.0.1:33902)
0924 04:07:06.270562 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:06.270617 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:06.270754 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:06.271364 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:06.271370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:06.271394 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:06.271544 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.271552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.272971 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.272974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.274254 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.274264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.275579 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.275583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.275663 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.275683 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:07:06.275747 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:07:06.276414 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":84,"threads_started":1}
I20260924 04:07:06.279181 19332 tablet_service.cc:1467] Processing CreateTablet for tablet d3f0561b8a754bdca2b77e54d78489a5 (DEFAULT_TABLE table=test-table [id=c445ad7cbf66434cb0b5ef6e215f056f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:06.279304 19463 tablet_service.cc:1467] Processing CreateTablet for tablet d3f0561b8a754bdca2b77e54d78489a5 (DEFAULT_TABLE table=test-table [id=c445ad7cbf66434cb0b5ef6e215f056f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:06.279304 19201 tablet_service.cc:1467] Processing CreateTablet for tablet d3f0561b8a754bdca2b77e54d78489a5 (DEFAULT_TABLE table=test-table [id=c445ad7cbf66434cb0b5ef6e215f056f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:06.280217 19332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f0561b8a754bdca2b77e54d78489a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:06.280354 19201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f0561b8a754bdca2b77e54d78489a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:06.280354 19463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f0561b8a754bdca2b77e54d78489a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:06.287602 19548 tablet_bootstrap.cc:492] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635: Bootstrap starting.
I20260924 04:07:06.288359 19546 tablet_bootstrap.cc:492] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: Bootstrap starting.
I20260924 04:07:06.289824 19548 tablet_bootstrap.cc:654] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:06.290118 19546 tablet_bootstrap.cc:654] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:06.290643 19548 log.cc:824] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:06.290769 19546 log.cc:824] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:06.291378 19547 tablet_bootstrap.cc:492] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Bootstrap starting.
I20260924 04:07:06.292733 19548 tablet_bootstrap.cc:492] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635: No bootstrap required, opened a new log
I20260924 04:07:06.292940 19548 ts_tablet_manager.cc:1397] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:07:06.293360 19546 tablet_bootstrap.cc:492] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: No bootstrap required, opened a new log
I20260924 04:07:06.293519 19547 tablet_bootstrap.cc:654] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:06.293563 19546 ts_tablet_manager.cc:1397] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:07:06.294188 19547 log.cc:824] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:06.295779 19547 tablet_bootstrap.cc:492] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: No bootstrap required, opened a new log
I20260924 04:07:06.295993 19547 ts_tablet_manager.cc:1397] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:07:06.297386 19548 raft_consensus.cc:348] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.297674 19548 raft_consensus.cc:374] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:06.297746 19548 raft_consensus.cc:729] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca3bd35ae2c433c9fa436db407cf635, State: Initialized, Role: FOLLOWER
I20260924 04:07:06.297613 19546 raft_consensus.cc:348] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.297873 19546 raft_consensus.cc:374] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:06.297921 19546 raft_consensus.cc:729] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ed304ee72c4761927e5c21d23933ed, State: Initialized, Role: FOLLOWER
I20260924 04:07:06.298286 19548 consensus_queue.cc:260] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.298285 19546 consensus_queue.cc:260] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.298995 19398 heartbeater.cc:503] Master 127.0.98.62:36009 was elected leader, sending a full tablet report...
I20260924 04:07:06.299345 19548 ts_tablet_manager.cc:1428] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:07:06.299335 19546 ts_tablet_manager.cc:1428] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:07:06.299472 19547 raft_consensus.cc:348] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.299705 19267 heartbeater.cc:503] Master 127.0.98.62:36009 was elected leader, sending a full tablet report...
I20260924 04:07:06.299714 19547 raft_consensus.cc:374] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:06.299768 19547 raft_consensus.cc:729] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38444bfdc784ceba94d0285d100327c, State: Initialized, Role: FOLLOWER
I20260924 04:07:06.300267 19547 consensus_queue.cc:260] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.301281 19547 ts_tablet_manager.cc:1428] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:07:06.301635 19529 heartbeater.cc:503] Master 127.0.98.62:36009 was elected leader, sending a full tablet report...
W20260924 04:07:06.365072 19268 tablet.cc:2369] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:07:06.472970 19553 raft_consensus.cc:482] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:07:06.473198 19553 raft_consensus.cc:504] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.474474 19553 leader_election.cc:288] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a38444bfdc784ceba94d0285d100327c (127.0.98.3:37327), 18ed304ee72c4761927e5c21d23933ed (127.0.98.2:42207)
I20260924 04:07:06.479672 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.474794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44913 (local address 127.0.98.3:37327)
0924 04:07:06.474920 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.474924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.475185 (+ 261us) server_negotiation.cc:407] Connection header received
0924 04:07:06.475328 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.475331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.475384 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.475464 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:06.475819 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.476634 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.478402 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.479035 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.479156 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.479221 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.479455 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.479556 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:07:06.479672 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.474857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60255 (local address 127.0.98.2:42207)
0924 04:07:06.474979 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.474984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.475252 (+ 268us) server_negotiation.cc:407] Connection header received
0924 04:07:06.475367 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.475371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.475416 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.475484 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:06.475819 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.476648 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.478402 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.479083 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.479175 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.479221 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.479455 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.479535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:07:06.479771 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.474762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42207 (local address 127.0.98.1:60255)
0924 04:07:06.475176 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:06.475187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:06.475260 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:06.475500 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:06.475503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:06.475530 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:06.475699 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.475706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.476756 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.476760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.478235 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.478247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.479229 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.479232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.479312 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.479345 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:07:06.479422 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:07:06.479533 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":191,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260924 04:07:06.479770 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.474711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37327 (local address 127.0.98.1:44913)
0924 04:07:06.475077 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:06.475091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:06.475215 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:06.475500 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:06.475503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:06.475535 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:06.475699 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.475706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.476756 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.476760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.478235 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:06.478247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.479183 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.479187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:06.479293 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.479356 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 04:07:06.479422 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:07:06.479534 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":98,"spinlock_wait_cycles":7424,"thread_start_us":87,"threads_started":1}
I20260924 04:07:06.480511 19483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f0561b8a754bdca2b77e54d78489a5" candidate_uuid: "7ca3bd35ae2c433c9fa436db407cf635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38444bfdc784ceba94d0285d100327c" is_pre_election: true
I20260924 04:07:06.480526 19352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f0561b8a754bdca2b77e54d78489a5" candidate_uuid: "7ca3bd35ae2c433c9fa436db407cf635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ed304ee72c4761927e5c21d23933ed" is_pre_election: true
I20260924 04:07:06.480818 19483 raft_consensus.cc:2454] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ca3bd35ae2c433c9fa436db407cf635 in term 0.
I20260924 04:07:06.480816 19352 raft_consensus.cc:2454] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ca3bd35ae2c433c9fa436db407cf635 in term 0.
I20260924 04:07:06.481305 19157 leader_election.cc:302] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ca3bd35ae2c433c9fa436db407cf635, a38444bfdc784ceba94d0285d100327c; no voters:
I20260924 04:07:06.481580 19553 raft_consensus.cc:2781] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:07:06.481673 19553 raft_consensus.cc:482] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:07:06.481721 19553 raft_consensus.cc:3037] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:07:06.482383 19530 tablet.cc:2369] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:07:06.482949 19553 raft_consensus.cc:504] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.483330 19553 leader_election.cc:288] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [CANDIDATE]: Term 1 election: Requested vote from peers a38444bfdc784ceba94d0285d100327c (127.0.98.3:37327), 18ed304ee72c4761927e5c21d23933ed (127.0.98.2:42207)
I20260924 04:07:06.483705 19352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f0561b8a754bdca2b77e54d78489a5" candidate_uuid: "7ca3bd35ae2c433c9fa436db407cf635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ed304ee72c4761927e5c21d23933ed"
I20260924 04:07:06.483835 19352 raft_consensus.cc:3037] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:06.484041 19483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f0561b8a754bdca2b77e54d78489a5" candidate_uuid: "7ca3bd35ae2c433c9fa436db407cf635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38444bfdc784ceba94d0285d100327c"
I20260924 04:07:06.484154 19483 raft_consensus.cc:3037] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:06.484911 19352 raft_consensus.cc:2454] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ca3bd35ae2c433c9fa436db407cf635 in term 1.
I20260924 04:07:06.485174 19483 raft_consensus.cc:2454] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ca3bd35ae2c433c9fa436db407cf635 in term 1.
I20260924 04:07:06.485268 19154 leader_election.cc:302] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed304ee72c4761927e5c21d23933ed, 7ca3bd35ae2c433c9fa436db407cf635; no voters:
I20260924 04:07:06.485445 19553 raft_consensus.cc:2781] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:06.485667 19553 raft_consensus.cc:686] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [term 1 LEADER]: Becoming Leader. State: Replica: 7ca3bd35ae2c433c9fa436db407cf635, State: Running, Role: LEADER
I20260924 04:07:06.485987 19553 consensus_queue.cc:237] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } }
I20260924 04:07:06.489142 19079 catalog_manager.cc:5495] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ca3bd35ae2c433c9fa436db407cf635 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7ca3bd35ae2c433c9fa436db407cf635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ca3bd35ae2c433c9fa436db407cf635" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46525 } health_report { overall_health: HEALTHY } } }
I20260924 04:07:06.513953 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.510329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.0.98.1:46525)
0924 04:07:06.510448 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:06.510452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:06.510467 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:07:06.510609 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:06.510612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:06.510674 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:06.510771 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:06.511015 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.511514 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.512459 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:06.512672 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:06.512759 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:06.513078 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:06.513511 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:06.513645 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:06.513778 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:07:06.513823 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:07:06.520188 19483 raft_consensus.cc:1261] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 1 FOLLOWER]: Refusing update from remote peer 7ca3bd35ae2c433c9fa436db407cf635: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:06.520188 19352 raft_consensus.cc:1261] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 1 FOLLOWER]: Refusing update from remote peer 7ca3bd35ae2c433c9fa436db407cf635: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:06.520889 19557 consensus_queue.cc:1035] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a38444bfdc784ceba94d0285d100327c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:06.521139 19553 consensus_queue.cc:1035] T d3f0561b8a754bdca2b77e54d78489a5 P 7ca3bd35ae2c433c9fa436db407cf635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18ed304ee72c4761927e5c21d23933ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:06.537891 19077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:06.538121 19077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:06.540927 19077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d68501285cc49ce9610d89d23867a35: Sending DeleteTablet for 3 replicas of tablet d3f0561b8a754bdca2b77e54d78489a5
I20260924 04:07:06.541893 19201 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f0561b8a754bdca2b77e54d78489a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:06 UTC) from {username='slave'} at 127.0.0.1:57384
I20260924 04:07:06.541893 19463 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f0561b8a754bdca2b77e54d78489a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:06 UTC) from {username='slave'} at 127.0.0.1:33902
I20260924 04:07:06.542471 19332 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f0561b8a754bdca2b77e54d78489a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:06 UTC) from {username='slave'} at 127.0.0.1:55152
I20260924 04:07:06.542946 19572 tablet_replica.cc:330] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: stopping tablet replica
I20260924 04:07:06.543566 19572 raft_consensus.cc:2229] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:06.543756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 19139
I20260924 04:07:06.544804 19572 raft_consensus.cc:2258] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:06.546432 19572 ts_tablet_manager.cc:1905] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:06.548157 19571 tablet_replica.cc:330] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: stopping tablet replica
I20260924 04:07:06.548486 19571 raft_consensus.cc:2229] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:06.548813 19571 raft_consensus.cc:2258] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:06.549369 19572 ts_tablet_manager.cc:1918] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:06.549468 19572 log.cc:1196] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/d3f0561b8a754bdca2b77e54d78489a5
I20260924 04:07:06.549769 19572 ts_tablet_manager.cc:1939] T d3f0561b8a754bdca2b77e54d78489a5 P 18ed304ee72c4761927e5c21d23933ed: Deleting consensus metadata
I20260924 04:07:06.550273 19571 ts_tablet_manager.cc:1905] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:06.551139 19063 catalog_manager.cc:4841] TS 18ed304ee72c4761927e5c21d23933ed (127.0.98.2:42207): tablet d3f0561b8a754bdca2b77e54d78489a5 (table test-table [id=c445ad7cbf66434cb0b5ef6e215f056f]) successfully deleted
W20260924 04:07:06.552030 19063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46525 (error 108)
I20260924 04:07:06.552647 19571 ts_tablet_manager.cc:1918] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:06.552747 19571 log.cc:1196] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/d3f0561b8a754bdca2b77e54d78489a5
I20260924 04:07:06.552855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 19270
I20260924 04:07:06.553048 19571 ts_tablet_manager.cc:1939] T d3f0561b8a754bdca2b77e54d78489a5 P a38444bfdc784ceba94d0285d100327c: Deleting consensus metadata
I20260924 04:07:06.553210 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:06.552806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46525 (local address 127.0.0.1:57396)
0924 04:07:06.553004 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:06.553110 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 04:07:06.554363 19066 catalog_manager.cc:4841] TS a38444bfdc784ceba94d0285d100327c (127.0.98.3:37327): tablet d3f0561b8a754bdca2b77e54d78489a5 (table test-table [id=c445ad7cbf66434cb0b5ef6e215f056f]) successfully deleted
W20260924 04:07:06.556162 19063 catalog_manager.cc:4584] TS 7ca3bd35ae2c433c9fa436db407cf635 (127.0.98.1:46525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3f0561b8a754bdca2b77e54d78489a5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46525: connect: Connection refused (error 111)
I20260924 04:07:06.558483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 19401
I20260924 04:07:06.563524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 19048
2026-09-24T04:07:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:06 dist-test-slave-zcsk krb5kdc[19576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:06 dist-test-slave-zcsk krb5kdc[19576](info): set up 2 sockets
Sep 24 04:07:06 dist-test-slave-zcsk krb5kdc[19576](info): commencing operation
krb5kdc: starting...
W20260924 04:07:07.531698 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:07 dist-test-slave-zcsk krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:07.651147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:07.761394 19589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:07.761687 19589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:07.761745 19589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:07.766237 19589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:07.766316 19589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:07.766345 19589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:07.766366 19589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:07.766387 19589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:07.771926 19589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:07.773057 19589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:07.774086 19589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:07.782788 19589 server_base.cc:1034] running on GCE node
W20260924 04:07:07.782774 19594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:07.782739 19597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:07.782740 19595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:07.783602 19589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:07.784789 19589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:07.785943 19589 hybrid_clock.cc:648] HybridClock initialized: now 1790222827785912 us; error 46 us; skew 500 ppm
Sep 24 04:07:07 dist-test-slave-zcsk krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:07.788625 19589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:07.789489 19589 webserver.cc:460] Webserver started at http://127.0.98.62:46043/ using document root <none> and password file <none>
I20260924 04:07:07.790025 19589 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:07.790089 19589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:07.790289 19589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:07.792384 19589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "3df3654639be4d6194b7b948385bb433"
format_stamp: "Formatted at 2026-09-24 04:07:07 on dist-test-slave-zcsk"
server_key: "4d6f1d48af7ab6d8bf1637b3e38c126b"
server_key_iv: "db471576d49073d7166a7808b6a2df18"
server_key_version: "encryptionkey@0"
I20260924 04:07:07.792958 19589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "3df3654639be4d6194b7b948385bb433"
format_stamp: "Formatted at 2026-09-24 04:07:07 on dist-test-slave-zcsk"
server_key: "4d6f1d48af7ab6d8bf1637b3e38c126b"
server_key_iv: "db471576d49073d7166a7808b6a2df18"
server_key_version: "encryptionkey@0"
I20260924 04:07:07.796911 19589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:07:07.799412 19604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:07.800840 19589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:07:07.800977 19589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "3df3654639be4d6194b7b948385bb433"
format_stamp: "Formatted at 2026-09-24 04:07:07 on dist-test-slave-zcsk"
server_key: "4d6f1d48af7ab6d8bf1637b3e38c126b"
server_key_iv: "db471576d49073d7166a7808b6a2df18"
server_key_version: "encryptionkey@0"
I20260924 04:07:07.801100 19589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:07.826336 19589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:07.829797 19589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:07.829984 19589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:07.836428 19589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39897
I20260924 04:07:07.836517 19656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39897 every 8 connection(s)
I20260924 04:07:07.837473 19589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:07.840236 19657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:07.845670 19657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433: Bootstrap starting.
I20260924 04:07:07.847476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 19589
I20260924 04:07:07.847610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:07.847877 392 external_mini_cluster.cc:1428] Setting key 6745376285509cf2953c1d99c9a63841
I20260924 04:07:07.848351 19657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:07.849184 19657 log.cc:824] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:07.851094 19657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433: No bootstrap required, opened a new log
I20260924 04:07:07.854487 19657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df3654639be4d6194b7b948385bb433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39897 } }
I20260924 04:07:07.854727 19657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:07.854784 19657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df3654639be4d6194b7b948385bb433, State: Initialized, Role: FOLLOWER
I20260924 04:07:07.855202 19657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df3654639be4d6194b7b948385bb433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39897 } }
I20260924 04:07:07.855329 19657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:07.855392 19657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:07.855487 19657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 04:07:07 dist-test-slave-zcsk krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:07.856463 19657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df3654639be4d6194b7b948385bb433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39897 } }
I20260924 04:07:07.856846 19657 leader_election.cc:302] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3df3654639be4d6194b7b948385bb433; no voters:
I20260924 04:07:07.857193 19657 leader_election.cc:288] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:07.857350 19662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:07.857592 19662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [term 1 LEADER]: Becoming Leader. State: Replica: 3df3654639be4d6194b7b948385bb433, State: Running, Role: LEADER
I20260924 04:07:07.857919 19662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df3654639be4d6194b7b948385bb433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39897 } }
I20260924 04:07:07.858592 19657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:07.858994 19662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3df3654639be4d6194b7b948385bb433. Latest consensus state: current_term: 1 leader_uuid: "3df3654639be4d6194b7b948385bb433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df3654639be4d6194b7b948385bb433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39897 } } }
I20260924 04:07:07.859109 19662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:07.859467 19666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:07.859781 19663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3df3654639be4d6194b7b948385bb433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df3654639be4d6194b7b948385bb433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39897 } } }
I20260924 04:07:07.859956 19663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:07.862391 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:07.849051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.0.98.62:39897)
0924 04:07:07.849528 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:07.849544 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:07.849580 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:07:07.850236 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:07.850254 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:07.850602 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:07.850924 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:07.851710 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:07.852761 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:07.853609 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:07.854091 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:07.854244 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:07.856736 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:07.856743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:07.856760 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:07.856793 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:07.858846 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:07.859391 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:07.859397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:07.859403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:07.859558 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:07.859887 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:07.859890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:07.859892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:07.860484 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:07.860719 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:07.861054 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 04:07:07.861431 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":136,"threads_started":1}
I20260924 04:07:07.862829 19666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:07.870174 19666 catalog_manager.cc:1347] Generated new cluster ID: b125834886f4446c84c5c5b23d97aba7
I20260924 04:07:07.870255 19666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:07.874899 19666 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:07.876231 19666 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:07.884590 19666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433: Generated new TSK 0
I20260924 04:07:07.885442 19666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:07:07.912617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39897
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:08.025344 19683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:08.025642 19683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:08.025688 19683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:08.029912 19683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:08.029981 19683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:08.030055 19683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:07:08.035363 19683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:08.036448 19683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:08.037494 19683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:08.046909 19691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:08.046927 19689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:08.046876 19688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:08.047145 19683 server_base.cc:1034] running on GCE node
I20260924 04:07:08.047557 19683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:08.048295 19683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:08.049461 19683 hybrid_clock.cc:648] HybridClock initialized: now 1790222828049431 us; error 44 us; skew 500 ppm
Sep 24 04:07:08 dist-test-slave-zcsk krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:08.052322 19683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:07:08.053273 19683 webserver.cc:460] Webserver started at http://127.0.98.1:34005/ using document root <none> and password file <none>
I20260924 04:07:08.053828 19683 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:08.053894 19683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:08.054098 19683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:08.056236 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "4d96e9ed92d64cc387a9585c9fddc9af"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "2ce4e723093969738964803e035e62ae"
server_key_iv: "c1c0a9f6a132e829f046ce9e21c102f2"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.056780 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4d96e9ed92d64cc387a9585c9fddc9af"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "2ce4e723093969738964803e035e62ae"
server_key_iv: "c1c0a9f6a132e829f046ce9e21c102f2"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.060748 19683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:07:08.063339 19698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.064644 19683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:08.064754 19683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "4d96e9ed92d64cc387a9585c9fddc9af"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "2ce4e723093969738964803e035e62ae"
server_key_iv: "c1c0a9f6a132e829f046ce9e21c102f2"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.064850 19683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:08.099004 19683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:08.101861 19683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:08.102042 19683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:08.102622 19683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:08.103434 19683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:08.103482 19683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.103538 19683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:08.103564 19683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.112057 19683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46171
I20260924 04:07:08.112078 19811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46171 every 8 connection(s)
I20260924 04:07:08.112917 19683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:07:08.119076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 19683
I20260924 04:07:08.119233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:07:08.119555 392 external_mini_cluster.cc:1428] Setting key 06cecd0923134359a34eaa1429744884
Sep 24 04:07:08 dist-test-slave-zcsk krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:08.127630 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.114749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50783 (local address 127.0.98.62:39897)
0924 04:07:08.114881 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.114885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.115727 (+ 842us) server_negotiation.cc:407] Connection header received
0924 04:07:08.117027 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.117030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.117077 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.117139 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:08.118375 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.119181 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.120512 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.120746 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.120850 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.123969 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:08.123973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:08.123975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:08.124004 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:08.125488 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:08.125995 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:08.125999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:08.126001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:08.126101 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:08.126393 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:08.126396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:08.126398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:08.126647 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:08.126752 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.127294 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.127440 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:07:08.128450 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.115138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39897 (local address 127.0.98.1:50783)
0924 04:07:08.115569 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.115613 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:08.116802 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:08.117303 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:08.117311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:08.117683 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:08.118234 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.118248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.119331 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.119335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.120392 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.120402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.120892 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.120894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.121089 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.121745 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:08.121771 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:08.123752 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:08.125627 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:08.125632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:08.125646 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:08.125899 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:08.126183 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:08.126185 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:08.126186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:08.126315 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:08.126755 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:08.126761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:08.127057 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 04:07:08.127224 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 04:07:08.127501 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:07:08.129741 19812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39897
I20260924 04:07:08.129995 19812 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:08.130524 19812 heartbeater.cc:511] Master 127.0.98.62:39897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:07:08.132175 19621 ts_manager.cc:194] Registered new tserver with Master: 4d96e9ed92d64cc387a9585c9fddc9af (127.0.98.1:46171)
I20260924 04:07:08.133395 19621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:07:08.147076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39897
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:08.258145 19817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:08.258482 19817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:08.258546 19817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:08.262995 19817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:08.263079 19817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:08.263171 19817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:07:08.268848 19817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:08.269956 19817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:08.271010 19817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:08.280674 19822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:08.280668 19825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:08.280717 19817 server_base.cc:1034] running on GCE node
W20260924 04:07:08.280668 19823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:08.281236 19817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:08.281869 19817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:08.283047 19817 hybrid_clock.cc:648] HybridClock initialized: now 1790222828283017 us; error 42 us; skew 500 ppm
Sep 24 04:07:08 dist-test-slave-zcsk krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:08.285697 19817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:07:08.286561 19817 webserver.cc:460] Webserver started at http://127.0.98.2:34479/ using document root <none> and password file <none>
I20260924 04:07:08.287143 19817 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:08.287209 19817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:08.287417 19817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:08.289505 19817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c02c1f0275f4dc6af701f1fb58469cc"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "2aaf8e9a8dbe2de783b2ef5973909a78"
server_key_iv: "f7a977e5a42c70b89aada4cc569d09ec"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.290035 19817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c02c1f0275f4dc6af701f1fb58469cc"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "2aaf8e9a8dbe2de783b2ef5973909a78"
server_key_iv: "f7a977e5a42c70b89aada4cc569d09ec"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.293790 19817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:07:08.296200 19832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.297449 19817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:07:08.297560 19817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "8c02c1f0275f4dc6af701f1fb58469cc"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "2aaf8e9a8dbe2de783b2ef5973909a78"
server_key_iv: "f7a977e5a42c70b89aada4cc569d09ec"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.297667 19817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:08.333140 19817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:08.336074 19817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:08.336246 19817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:08.336862 19817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:08.337682 19817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:08.337729 19817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.337769 19817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:08.337806 19817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.346437 19817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42739
I20260924 04:07:08.346510 19945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42739 every 8 connection(s)
I20260924 04:07:08.347409 19817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:07:08.354297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 19817
I20260924 04:07:08.354413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:07:08.354689 392 external_mini_cluster.cc:1428] Setting key 0085a4b0a79407cda998c57359bab052
Sep 24 04:07:08 dist-test-slave-zcsk krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:08.362726 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.349381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38543 (local address 127.0.98.62:39897)
0924 04:07:08.349539 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.349542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.350505 (+ 963us) server_negotiation.cc:407] Connection header received
0924 04:07:08.351830 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.351833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.351885 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.352007 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:08.353350 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.353887 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.355303 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.355579 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.355674 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.358760 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:08.358763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:08.358766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:08.358795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:08.360485 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:08.361111 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:08.361114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:08.361116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:08.361196 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:08.361552 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:08.361556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:08.361558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:08.361777 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:08.361882 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.362418 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.362561 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:07:08.363623 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.349772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39897 (local address 127.0.98.2:38543)
0924 04:07:08.350331 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.350379 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:08.351607 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:08.352197 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:08.352206 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:08.352615 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:08.353187 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.353202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.354026 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.354029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.355152 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.355165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.355738 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.355743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.355955 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.356759 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:08.356785 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:08.358540 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:08.360597 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:08.360604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:08.360624 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:08.361008 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:08.361296 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:08.361299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:08.361301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:08.361458 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:08.361889 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:08.361896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:08.362176 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 04:07:08.362358 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:07:08.362724 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":149,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:07:08.364931 19946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39897
I20260924 04:07:08.365193 19946 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:08.365689 19946 heartbeater.cc:511] Master 127.0.98.62:39897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:07:08.366911 19621 ts_manager.cc:194] Registered new tserver with Master: 8c02c1f0275f4dc6af701f1fb58469cc (127.0.98.2:42739)
I20260924 04:07:08.367679 19621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:07:08.384035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39897
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:08.494674 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:08.494967 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:08.495038 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:08.499274 19951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:08.499353 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:08.499440 19951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:07:08.505067 19951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:08.506225 19951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:08.507283 19951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:08.516706 19956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:08.516712 19959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:08.516886 19957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:08.516904 19951 server_base.cc:1034] running on GCE node
I20260924 04:07:08.517395 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:08.518038 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:08.519207 19951 hybrid_clock.cc:648] HybridClock initialized: now 1790222828519175 us; error 43 us; skew 500 ppm
Sep 24 04:07:08 dist-test-slave-zcsk krb5kdc[19576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:08.521889 19951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:07:08.522805 19951 webserver.cc:460] Webserver started at http://127.0.98.3:37891/ using document root <none> and password file <none>
I20260924 04:07:08.523350 19951 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:08.523415 19951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:08.523613 19951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:08.525804 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "cf828bf1d3a842708a37b15e11b1de20"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "8fba90bea79b76d3e9d020bed65b3d25"
server_key_iv: "67ea38e9043621939b858daa627c9e2d"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.526369 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cf828bf1d3a842708a37b15e11b1de20"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "8fba90bea79b76d3e9d020bed65b3d25"
server_key_iv: "67ea38e9043621939b858daa627c9e2d"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.530270 19951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:07:08.532855 19966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.534126 19951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:08.534231 19951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "cf828bf1d3a842708a37b15e11b1de20"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "8fba90bea79b76d3e9d020bed65b3d25"
server_key_iv: "67ea38e9043621939b858daa627c9e2d"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.534341 19951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:08.553483 19951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:08.557057 19951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:08.557214 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:08.557760 19951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:08.558568 19951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:08.558615 19951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.558655 19951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:08.558673 19951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.567377 19951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40395
I20260924 04:07:08.567416 20079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40395 every 8 connection(s)
I20260924 04:07:08.568440 19951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:07:08.570361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 19951
I20260924 04:07:08.570480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:07:08.570753 392 external_mini_cluster.cc:1428] Setting key a590ba948db15cf9c3fa0a94fc71170f
Sep 24 04:07:08 dist-test-slave-zcsk krb5kdc[19576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:08.581691 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.570371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41575 (local address 127.0.98.62:39897)
0924 04:07:08.570531 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.570536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.571380 (+ 844us) server_negotiation.cc:407] Connection header received
0924 04:07:08.572743 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.572747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.572799 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.572862 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:08.574032 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.574564 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.575657 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.575875 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.576069 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.578430 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:08.578434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:08.578436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:08.578463 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:08.579629 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:08.580207 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:08.580211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:08.580213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:08.580293 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:08.580616 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:08.580619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:08.580621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:08.580805 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:08.580889 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.581399 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.581523 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:07:08.582557 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.570741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39897 (local address 127.0.98.3:41575)
0924 04:07:08.571223 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.571278 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:08.572561 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:08.573018 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:08.573025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:08.573381 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:08.573893 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.573907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.574659 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.574661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.575550 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.575558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.576099 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.576102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.576248 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.576818 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:08.576835 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:08.578255 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:08.579761 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:08.579767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:08.579784 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:08.580112 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:08.580373 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:08.580376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:08.580378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:08.580528 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:08.580904 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:08.580911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:08.581170 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 04:07:08.581343 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 04:07:08.581694 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260924 04:07:08.583808 20080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39897
I20260924 04:07:08.584100 20080 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:08.584645 20080 heartbeater.cc:511] Master 127.0.98.62:39897 requested a full tablet report, sending...
I20260924 04:07:08.585625 19616 ts_manager.cc:194] Registered new tserver with Master: cf828bf1d3a842708a37b15e11b1de20 (127.0.98.3:40395)
I20260924 04:07:08.586213 19616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41575
I20260924 04:07:08.596167 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:07:08.604882 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.598041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.0.98.62:39897)
0924 04:07:08.598170 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.598175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.598336 (+ 161us) server_negotiation.cc:407] Connection header received
0924 04:07:08.598447 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.598450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.598493 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.598555 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:08.599166 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.599669 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.600530 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.600813 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.600894 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.602237 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:08.602240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:08.602242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:08.602268 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:08.603422 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:08.603775 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:08.603777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:08.603779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:08.603847 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:08.604115 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:08.604118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:08.604119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:08.604334 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:08.604439 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.604610 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.604717 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:07:08.607869 19616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:07:08.610894 19616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:07:08.625262 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.619247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40395 (local address 127.0.0.1:48542)
0924 04:07:08.619618 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.619670 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:08.619797 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:08.620815 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:08.620821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:08.620845 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:08.620998 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.621005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.622227 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.622230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.623518 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.623528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.624724 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.624727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.624831 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.624857 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:07:08.624922 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:07:08.625078 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":21,"thread_start_us":77,"threads_started":1}
I20260924 04:07:08.625397 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.620200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.0.98.2:42739)
0924 04:07:08.621146 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.621157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.621177 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:07:08.621243 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.621248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.621419 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.621558 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:08.621751 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.622771 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.624040 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.624827 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.624931 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.625071 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.625138 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.625242 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":104,"threads_started":1}
I20260924 04:07:08.625429 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.619391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.0.98.3:40395)
0924 04:07:08.620355 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.620366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.620389 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:07:08.620476 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.620481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.620654 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.621128 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:08.621257 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.622110 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.623664 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.624584 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.625048 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.625170 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.625219 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.625299 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":198,"threads_started":1}
I20260924 04:07:08.625581 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.619467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46171 (local address 127.0.0.1:42734)
0924 04:07:08.620026 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.620038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:08.620116 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:08.620939 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:08.620942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:08.620956 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:08.621067 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.621073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.622638 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.622642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.623713 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.623721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.624677 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.624681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.624787 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.624822 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:07:08.624928 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:07:08.625470 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260924 04:07:08.625892 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.620092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.0.98.1:46171)
0924 04:07:08.620486 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.620499 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.620520 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:07:08.620594 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.620598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.620796 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.621206 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:08.621351 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.622489 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.623830 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.624566 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.624664 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.624821 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.625668 (+ 847us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.625778 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
I20260924 04:07:08.625262 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.620351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42739 (local address 127.0.0.1:52016)
0924 04:07:08.620804 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.620817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:08.620908 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:08.621558 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:08.621561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:08.621579 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:08.621655 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.621660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.622882 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.622885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.623926 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.623936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.624934 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.624938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.625024 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.625043 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:07:08.625090 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:07:08.625143 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"spinlock_wait_cycles":896,"thread_start_us":81,"threads_started":1}
I20260924 04:07:08.628820 19880 tablet_service.cc:1467] Processing CreateTablet for tablet 11001c35b1fe47de8f05d6b3e7886d27 (DEFAULT_TABLE table=test-table [id=a63d750ae65a42d8aed6565586c60731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:08.628849 20014 tablet_service.cc:1467] Processing CreateTablet for tablet 11001c35b1fe47de8f05d6b3e7886d27 (DEFAULT_TABLE table=test-table [id=a63d750ae65a42d8aed6565586c60731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:08.628849 19746 tablet_service.cc:1467] Processing CreateTablet for tablet 11001c35b1fe47de8f05d6b3e7886d27 (DEFAULT_TABLE table=test-table [id=a63d750ae65a42d8aed6565586c60731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:08.629842 19880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11001c35b1fe47de8f05d6b3e7886d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:08.629859 19746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11001c35b1fe47de8f05d6b3e7886d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:08.629859 20014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11001c35b1fe47de8f05d6b3e7886d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:08.636672 20097 tablet_bootstrap.cc:492] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: Bootstrap starting.
I20260924 04:07:08.637601 20099 tablet_bootstrap.cc:492] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: Bootstrap starting.
I20260924 04:07:08.639315 20097 tablet_bootstrap.cc:654] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:08.639473 20099 tablet_bootstrap.cc:654] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:08.640172 20097 log.cc:824] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:08.640460 20099 log.cc:824] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:08.640890 20098 tablet_bootstrap.cc:492] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af: Bootstrap starting.
I20260924 04:07:08.642236 20097 tablet_bootstrap.cc:492] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: No bootstrap required, opened a new log
I20260924 04:07:08.642439 20097 ts_tablet_manager.cc:1397] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:07:08.643072 20098 tablet_bootstrap.cc:654] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:08.643786 20098 log.cc:824] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:08.645608 20099 tablet_bootstrap.cc:492] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: No bootstrap required, opened a new log
I20260924 04:07:08.645800 20098 tablet_bootstrap.cc:492] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af: No bootstrap required, opened a new log
I20260924 04:07:08.645831 20099 ts_tablet_manager.cc:1397] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260924 04:07:08.645637 20097 raft_consensus.cc:348] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.645922 20097 raft_consensus.cc:374] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:08.645977 20097 raft_consensus.cc:729] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c02c1f0275f4dc6af701f1fb58469cc, State: Initialized, Role: FOLLOWER
I20260924 04:07:08.645984 20098 ts_tablet_manager.cc:1397] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:07:08.646430 20097 consensus_queue.cc:260] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.647083 19946 heartbeater.cc:503] Master 127.0.98.62:39897 was elected leader, sending a full tablet report...
I20260924 04:07:08.647455 20097 ts_tablet_manager.cc:1428] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:07:08.649969 20099 raft_consensus.cc:348] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.650296 20099 raft_consensus.cc:374] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:08.650365 20099 raft_consensus.cc:729] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf828bf1d3a842708a37b15e11b1de20, State: Initialized, Role: FOLLOWER
I20260924 04:07:08.650835 20099 consensus_queue.cc:260] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.650748 20098 raft_consensus.cc:348] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.650997 20098 raft_consensus.cc:374] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:08.651054 20098 raft_consensus.cc:729] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d96e9ed92d64cc387a9585c9fddc9af, State: Initialized, Role: FOLLOWER
I20260924 04:07:08.651546 20080 heartbeater.cc:503] Master 127.0.98.62:39897 was elected leader, sending a full tablet report...
I20260924 04:07:08.651549 20098 consensus_queue.cc:260] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.651990 20099 ts_tablet_manager.cc:1428] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260924 04:07:08.652325 19812 heartbeater.cc:503] Master 127.0.98.62:39897 was elected leader, sending a full tablet report...
I20260924 04:07:08.652606 20098 ts_tablet_manager.cc:1428] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260924 04:07:08.681682 20103 raft_consensus.cc:482] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:07:08.681872 20103 raft_consensus.cc:504] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.683043 20103 leader_election.cc:288] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf828bf1d3a842708a37b15e11b1de20 (127.0.98.3:40395), 4d96e9ed92d64cc387a9585c9fddc9af (127.0.98.1:46171)
I20260924 04:07:08.686929 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.683270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46171 (local address 127.0.98.2:36253)
0924 04:07:08.683389 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.683416 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:08.683505 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:08.683747 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:08.683750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:08.683769 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:08.683877 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.683883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.684664 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.684666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.685801 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.685819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.686578 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.686581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.686661 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.686682 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:07:08.686737 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 04:07:08.686798 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:07:08.686973 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.683386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36253 (local address 127.0.98.1:46171)
0924 04:07:08.683502 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.683506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.683520 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:07:08.683599 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.683601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.683639 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.683712 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:08.684028 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.684565 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.685967 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.686473 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.686571 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.686637 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.686769 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.686845 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:07:08.687391 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.683362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48035 (local address 127.0.98.3:40395)
0924 04:07:08.683490 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.683494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.683981 (+ 487us) server_negotiation.cc:407] Connection header received
0924 04:07:08.684110 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.684112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.684155 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.684214 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:08.684485 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.685013 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.686266 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.686801 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.686897 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.686962 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.687209 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.687279 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:07:08.687743 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.683270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40395 (local address 127.0.98.2:48035)
0924 04:07:08.683856 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.683867 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:08.684015 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:08.684232 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:08.684234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:08.684249 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:08.684361 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.684367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.685118 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.685121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.686151 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:08.686159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.686964 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.686968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:08.687062 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.687084 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:07:08.687542 (+ 458us) client_negotiation.cc:240] Negotiation successful
0924 04:07:08.687617 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":306,"thread_start_us":108,"threads_started":1}
I20260924 04:07:08.687665 19766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11001c35b1fe47de8f05d6b3e7886d27" candidate_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" is_pre_election: true
I20260924 04:07:08.688048 19766 raft_consensus.cc:2454] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c02c1f0275f4dc6af701f1fb58469cc in term 0.
I20260924 04:07:08.688401 20034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11001c35b1fe47de8f05d6b3e7886d27" candidate_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf828bf1d3a842708a37b15e11b1de20" is_pre_election: true
I20260924 04:07:08.688527 19835 leader_election.cc:302] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d96e9ed92d64cc387a9585c9fddc9af, 8c02c1f0275f4dc6af701f1fb58469cc; no voters:
I20260924 04:07:08.688642 20034 raft_consensus.cc:2454] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c02c1f0275f4dc6af701f1fb58469cc in term 0.
I20260924 04:07:08.688772 20103 raft_consensus.cc:2781] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:07:08.688843 20103 raft_consensus.cc:482] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:07:08.688879 20103 raft_consensus.cc:3037] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:08.689915 20103 raft_consensus.cc:504] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.690248 20103 leader_election.cc:288] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [CANDIDATE]: Term 1 election: Requested vote from peers cf828bf1d3a842708a37b15e11b1de20 (127.0.98.3:40395), 4d96e9ed92d64cc387a9585c9fddc9af (127.0.98.1:46171)
I20260924 04:07:08.690588 19766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11001c35b1fe47de8f05d6b3e7886d27" candidate_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d96e9ed92d64cc387a9585c9fddc9af"
I20260924 04:07:08.690580 20034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11001c35b1fe47de8f05d6b3e7886d27" candidate_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf828bf1d3a842708a37b15e11b1de20"
I20260924 04:07:08.690706 20034 raft_consensus.cc:3037] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:08.690706 19766 raft_consensus.cc:3037] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:08.691978 20034 raft_consensus.cc:2454] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c02c1f0275f4dc6af701f1fb58469cc in term 1.
I20260924 04:07:08.691978 19766 raft_consensus.cc:2454] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c02c1f0275f4dc6af701f1fb58469cc in term 1.
I20260924 04:07:08.692260 19835 leader_election.cc:302] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d96e9ed92d64cc387a9585c9fddc9af, 8c02c1f0275f4dc6af701f1fb58469cc; no voters:
I20260924 04:07:08.692420 20103 raft_consensus.cc:2781] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:08.692634 20103 raft_consensus.cc:686] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 1 LEADER]: Becoming Leader. State: Replica: 8c02c1f0275f4dc6af701f1fb58469cc, State: Running, Role: LEADER
I20260924 04:07:08.692968 20103 consensus_queue.cc:237] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } }
I20260924 04:07:08.696411 19616 catalog_manager.cc:5495] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c02c1f0275f4dc6af701f1fb58469cc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c02c1f0275f4dc6af701f1fb58469cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42739 } health_report { overall_health: HEALTHY } } }
I20260924 04:07:08.724021 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.720493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.0.98.2:42739)
0924 04:07:08.720671 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.720676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.720702 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:07:08.720809 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.720811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.720867 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.720992 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:08.721217 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.721714 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.722556 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.722770 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.722856 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.723165 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:08.723601 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:08.723711 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.723780 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.723840 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:07:08.730021 20034 raft_consensus.cc:1261] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 1 FOLLOWER]: Refusing update from remote peer 8c02c1f0275f4dc6af701f1fb58469cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:08.730021 19766 raft_consensus.cc:1261] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af [term 1 FOLLOWER]: Refusing update from remote peer 8c02c1f0275f4dc6af701f1fb58469cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:08.730825 20107 consensus_queue.cc:1035] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d96e9ed92d64cc387a9585c9fddc9af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:08.731027 20103 consensus_queue.cc:1035] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf828bf1d3a842708a37b15e11b1de20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:08.739702 20110 mvcc.cc:205] Tried to move back new op lower bound from 7332752706470604800 to 7332752706329227264. Current Snapshot: MvccSnapshot[applied={T|T < 7332752706470604800}]
I20260924 04:07:08.739883 20112 mvcc.cc:205] Tried to move back new op lower bound from 7332752706470604800 to 7332752706329227264. Current Snapshot: MvccSnapshot[applied={T|T < 7332752706470604800}]
I20260924 04:07:08.747856 19616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:08.748061 19616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:08.750538 19616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3df3654639be4d6194b7b948385bb433: Sending DeleteTablet for 3 replicas of tablet 11001c35b1fe47de8f05d6b3e7886d27
I20260924 04:07:08.751422 19746 tablet_service.cc:1514] Processing DeleteTablet for tablet 11001c35b1fe47de8f05d6b3e7886d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42734
I20260924 04:07:08.751513 19880 tablet_service.cc:1514] Processing DeleteTablet for tablet 11001c35b1fe47de8f05d6b3e7886d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52016
I20260924 04:07:08.752000 20014 tablet_service.cc:1514] Processing DeleteTablet for tablet 11001c35b1fe47de8f05d6b3e7886d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48542
I20260924 04:07:08.752081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 19683
I20260924 04:07:08.752116 20120 tablet_replica.cc:330] T 11001c35b1fe47de8f05d6b3e7886d27 P 4d96e9ed92d64cc387a9585c9fddc9af: stopping tablet replica
I20260924 04:07:08.752872 20121 tablet_replica.cc:330] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: stopping tablet replica
I20260924 04:07:08.753033 20122 tablet_replica.cc:330] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: stopping tablet replica
I20260924 04:07:08.753194 20121 raft_consensus.cc:2229] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:07:08.753255 20122 raft_consensus.cc:2229] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:08.753551 20122 raft_consensus.cc:2258] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:08.753572 20121 raft_consensus.cc:2258] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:08.755118 20122 ts_tablet_manager.cc:1905] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:08.755118 20121 ts_tablet_manager.cc:1905] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:08.757819 20122 ts_tablet_manager.cc:1918] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:08.757819 20121 ts_tablet_manager.cc:1918] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:08.757929 20121 log.cc:1196] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/11001c35b1fe47de8f05d6b3e7886d27
I20260924 04:07:08.757937 20122 log.cc:1196] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/11001c35b1fe47de8f05d6b3e7886d27
I20260924 04:07:08.758262 20122 ts_tablet_manager.cc:1939] T 11001c35b1fe47de8f05d6b3e7886d27 P cf828bf1d3a842708a37b15e11b1de20: Deleting consensus metadata
I20260924 04:07:08.758262 20121 ts_tablet_manager.cc:1939] T 11001c35b1fe47de8f05d6b3e7886d27 P 8c02c1f0275f4dc6af701f1fb58469cc: Deleting consensus metadata
I20260924 04:07:08.759814 19605 catalog_manager.cc:4841] TS cf828bf1d3a842708a37b15e11b1de20 (127.0.98.3:40395): tablet 11001c35b1fe47de8f05d6b3e7886d27 (table test-table [id=a63d750ae65a42d8aed6565586c60731]) successfully deleted
I20260924 04:07:08.760077 19605 catalog_manager.cc:4841] TS 8c02c1f0275f4dc6af701f1fb58469cc (127.0.98.2:42739): tablet 11001c35b1fe47de8f05d6b3e7886d27 (table test-table [id=a63d750ae65a42d8aed6565586c60731]) successfully deleted
W20260924 04:07:08.761399 19607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46171 (error 108)
I20260924 04:07:08.762283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 19817
I20260924 04:07:08.762534 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.762232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46171 (local address 127.0.0.1:42742)
0924 04:07:08.762364 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:08.762451 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 04:07:08.763049 19607 catalog_manager.cc:4584] TS 4d96e9ed92d64cc387a9585c9fddc9af (127.0.98.1:46171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11001c35b1fe47de8f05d6b3e7886d27: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46171: connect: Connection refused (error 111)
I20260924 04:07:08.767544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 19951
I20260924 04:07:08.772230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 19589
2026-09-24T04:07:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2209 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T04:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:08Z Disabled control of system clock
I20260924 04:07:08.801333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45571
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:07:08.912047 20127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:08.912315 20127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:08.912350 20127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:08.916329 20127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:08.916384 20127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:08.916400 20127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:08.916415 20127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:08.916429 20127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:08.921327 20127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:08.922375 20127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:08.923748 20127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:08.932286 20133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:08.932286 20132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:08.932291 20135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:08.932327 20127 server_base.cc:1034] running on GCE node
I20260924 04:07:08.933135 20127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:08.934254 20127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:08.935434 20127 hybrid_clock.cc:648] HybridClock initialized: now 1790222828935414 us; error 34 us; skew 500 ppm
I20260924 04:07:08.937520 20127 webserver.cc:460] Webserver started at http://127.0.98.62:36225/ using document root <none> and password file <none>
I20260924 04:07:08.938095 20127 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:08.938164 20127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:08.938378 20127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:08.940726 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "feae1cbbcd694c37a570c191ef6a1881"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "663b40ea5f840dad761f718ec700ed86"
server_key_iv: "a2656e6666cac938d3cd4da76802ac42"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.941269 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "feae1cbbcd694c37a570c191ef6a1881"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "663b40ea5f840dad761f718ec700ed86"
server_key_iv: "a2656e6666cac938d3cd4da76802ac42"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.944909 20127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:07:08.947489 20141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:08.948858 20127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:08.949016 20127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "feae1cbbcd694c37a570c191ef6a1881"
format_stamp: "Formatted at 2026-09-24 04:07:08 on dist-test-slave-zcsk"
server_key: "663b40ea5f840dad761f718ec700ed86"
server_key_iv: "a2656e6666cac938d3cd4da76802ac42"
server_key_version: "encryptionkey@0"
I20260924 04:07:08.949126 20127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:08.967309 20127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:08.968145 20127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:08.968336 20127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:08.974953 20127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45571
I20260924 04:07:08.974958 20193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45571 every 8 connection(s)
I20260924 04:07:08.975950 20127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:08.977131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 20127
I20260924 04:07:08.977264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:08.977571 392 external_mini_cluster.cc:1428] Setting key 4c116ac075ae27875c355ba4ed2ac7ac
I20260924 04:07:08.979791 20194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:08.985301 20194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881: Bootstrap starting.
I20260924 04:07:08.986601 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:08.979084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.0.98.62:45571)
0924 04:07:08.979762 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:08.979777 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:08.979811 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:07:08.980519 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:08.980536 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:08.980942 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:08.981251 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:08.981672 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.982668 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.983521 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:08.984025 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:08.984161 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:08.984959 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:08.984964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:08.984976 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:08.985350 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:08.985420 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:08.985448 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:08.985528 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:08.985656 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 04:07:08.985911 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":137,"threads_started":1}
I20260924 04:07:08.987560 20194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:08.988307 20194 log.cc:824] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:08.990212 20194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881: No bootstrap required, opened a new log
I20260924 04:07:08.993790 20194 raft_consensus.cc:348] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae1cbbcd694c37a570c191ef6a1881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45571 } }
I20260924 04:07:08.994038 20194 raft_consensus.cc:374] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:08.994097 20194 raft_consensus.cc:729] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feae1cbbcd694c37a570c191ef6a1881, State: Initialized, Role: FOLLOWER
I20260924 04:07:08.994539 20194 consensus_queue.cc:260] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae1cbbcd694c37a570c191ef6a1881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45571 } }
I20260924 04:07:08.994695 20194 raft_consensus.cc:388] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:08.994763 20194 raft_consensus.cc:482] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:08.994867 20194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:08.995784 20194 raft_consensus.cc:504] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae1cbbcd694c37a570c191ef6a1881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45571 } }
I20260924 04:07:08.996147 20194 leader_election.cc:302] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feae1cbbcd694c37a570c191ef6a1881; no voters:
I20260924 04:07:08.996496 20194 leader_election.cc:288] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:08.996605 20199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:08.996829 20199 raft_consensus.cc:686] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [term 1 LEADER]: Becoming Leader. State: Replica: feae1cbbcd694c37a570c191ef6a1881, State: Running, Role: LEADER
I20260924 04:07:08.997150 20199 consensus_queue.cc:237] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae1cbbcd694c37a570c191ef6a1881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45571 } }
I20260924 04:07:08.997870 20194 sys_catalog.cc:564] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:08.998883 20200 sys_catalog.cc:455] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feae1cbbcd694c37a570c191ef6a1881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae1cbbcd694c37a570c191ef6a1881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45571 } } }
I20260924 04:07:08.998914 20201 sys_catalog.cc:455] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feae1cbbcd694c37a570c191ef6a1881. Latest consensus state: current_term: 1 leader_uuid: "feae1cbbcd694c37a570c191ef6a1881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae1cbbcd694c37a570c191ef6a1881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45571 } } }
I20260924 04:07:08.999135 20201 sys_catalog.cc:458] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:08.999168 20200 sys_catalog.cc:458] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:08.999611 20208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:09.002846 20208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:09.008534 20208 catalog_manager.cc:1347] Generated new cluster ID: eb8ac19f00b946f1b768b64ba069e026
I20260924 04:07:09.008620 20208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:09.043151 20208 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:09.044539 20208 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:09.052909 20208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881: Generated new TSK 0
I20260924 04:07:09.053664 20208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:09.057642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45571
--builtin_ntp_servers=127.0.98.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:07:09.179831 20218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:09.180147 20218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:09.180186 20218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:09.184309 20218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:09.184374 20218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:09.184437 20218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:07:09.189334 20218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:09.190519 20218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:09.191992 20218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:09.201877 20223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:09.201951 20226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:09.202453 20218 server_base.cc:1034] running on GCE node
W20260924 04:07:09.202826 20224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:09.203266 20218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:09.203975 20218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:09.205144 20218 hybrid_clock.cc:648] HybridClock initialized: now 1790222829205102 us; error 61 us; skew 500 ppm
I20260924 04:07:09.207098 20218 webserver.cc:460] Webserver started at http://127.0.98.1:44529/ using document root <none> and password file <none>
I20260924 04:07:09.207676 20218 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:09.207746 20218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:09.207996 20218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:09.210153 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/data/instance:
uuid: "5489bb91bd38429dbcbfab2238b6135f"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "f2b87e78723d3bd7c29ecfad2e319f79"
server_key_iv: "19c4977dc9c8732c14b64c6fdf578bf8"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.210721 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5489bb91bd38429dbcbfab2238b6135f"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "f2b87e78723d3bd7c29ecfad2e319f79"
server_key_iv: "19c4977dc9c8732c14b64c6fdf578bf8"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.214677 20218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:07:09.217427 20232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.218973 20218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:07:09.219105 20218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/wal
uuid: "5489bb91bd38429dbcbfab2238b6135f"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "f2b87e78723d3bd7c29ecfad2e319f79"
server_key_iv: "19c4977dc9c8732c14b64c6fdf578bf8"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.219224 20218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:09.241024 20218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:09.241696 20218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:09.241859 20218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:09.242409 20218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:09.243322 20218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:09.243369 20218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.243408 20218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:09.243429 20218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.252751 20218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39091
I20260924 04:07:09.252810 20345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39091 every 8 connection(s)
I20260924 04:07:09.253727 20218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:07:09.255169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 20218
I20260924 04:07:09.255296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:07:09.255571 392 external_mini_cluster.cc:1428] Setting key d8925452581711fde8b4e587041bb553
I20260924 04:07:09.257682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45571
--builtin_ntp_servers=127.0.98.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:07:09.264534 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.255792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34621 (local address 127.0.98.62:45571)
0924 04:07:09.255992 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.255997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.257226 (+ 1229us) server_negotiation.cc:407] Connection header received
0924 04:07:09.258705 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.258709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.258777 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.258874 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:09.260202 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.261061 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.262279 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.262628 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.262760 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.263974 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:09.263979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:09.263982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:09.263999 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:09.264027 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:09.264039 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:09.264120 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.264264 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.264362 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:07:09.265411 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.256262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45571 (local address 127.0.98.1:34621)
0924 04:07:09.257071 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.257128 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:09.258459 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:09.259119 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:09.259130 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:09.259260 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:09.260034 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.260052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.261204 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.261206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.262150 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.262159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.262800 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.262803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.262984 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.263650 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:09.263652 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:09.263698 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:09.263773 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:09.264121 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:09.264129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:09.264133 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:07:09.264240 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:07:09.264503 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":120,"threads_started":1}
I20260924 04:07:09.266693 20346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45571
I20260924 04:07:09.266942 20346 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:09.267475 20346 heartbeater.cc:511] Master 127.0.98.62:45571 requested a full tablet report, sending...
I20260924 04:07:09.268980 20158 ts_manager.cc:194] Registered new tserver with Master: 5489bb91bd38429dbcbfab2238b6135f (127.0.98.1:39091)
I20260924 04:07:09.270118 20158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34621
W20260924 04:07:09.369037 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:09.369338 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:09.369376 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:09.373474 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:09.373533 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:09.373591 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:07:09.378518 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:09.379614 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:09.381007 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:09.390300 20354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:09.390381 20349 server_base.cc:1034] running on GCE node
W20260924 04:07:09.390321 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:09.390312 20357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:09.390923 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:09.391593 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:09.392750 20349 hybrid_clock.cc:648] HybridClock initialized: now 1790222829392723 us; error 41 us; skew 500 ppm
I20260924 04:07:09.394666 20349 webserver.cc:460] Webserver started at http://127.0.98.2:35729/ using document root <none> and password file <none>
I20260924 04:07:09.395236 20349 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:09.395303 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:09.395519 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:09.397676 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/data/instance:
uuid: "f8e0ab815ab44792821a8c03f997b5b0"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "bb8bef3b1881b444be712021d6a5863a"
server_key_iv: "15b50c5fd2ea19c232947eb47ad99252"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.398283 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f8e0ab815ab44792821a8c03f997b5b0"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "bb8bef3b1881b444be712021d6a5863a"
server_key_iv: "15b50c5fd2ea19c232947eb47ad99252"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.402156 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 04:07:09.404654 20363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.405926 20349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 04:07:09.406050 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/wal
uuid: "f8e0ab815ab44792821a8c03f997b5b0"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "bb8bef3b1881b444be712021d6a5863a"
server_key_iv: "15b50c5fd2ea19c232947eb47ad99252"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.406160 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:09.418367 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:09.419082 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:09.419262 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:09.419852 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:09.420835 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:09.420892 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.420948 20349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:09.420974 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.429571 20349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43323
I20260924 04:07:09.429603 20476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43323 every 8 connection(s)
I20260924 04:07:09.430529 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:07:09.434599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 20349
I20260924 04:07:09.434803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:07:09.435115 392 external_mini_cluster.cc:1428] Setting key 91a1c51132ab9e6e945b0a0bfc8fac10
I20260924 04:07:09.437134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45571
--builtin_ntp_servers=127.0.98.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:07:09.440577 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.432428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43719 (local address 127.0.98.62:45571)
0924 04:07:09.432576 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.432580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.433583 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 04:07:09.435173 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.435177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.435239 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.435314 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:09.436367 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.436947 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.438516 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.438844 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.438957 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.440116 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:09.440119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:09.440122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:09.440137 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:09.440162 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:09.440172 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:09.440238 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.440364 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.440428 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:07:09.441447 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.432800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45571 (local address 127.0.98.2:43719)
0924 04:07:09.433456 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.433496 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:09.434941 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:09.435525 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:09.435533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:09.435623 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:09.436231 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.436245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.437078 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.437081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.438389 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.438401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.439026 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.439029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.439187 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.439797 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:09.439798 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:09.439853 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:09.439962 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:09.440247 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:09.440252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:09.440255 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:07:09.440353 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:07:09.440595 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
I20260924 04:07:09.442771 20477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45571
I20260924 04:07:09.443002 20477 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:09.443527 20477 heartbeater.cc:511] Master 127.0.98.62:45571 requested a full tablet report, sending...
I20260924 04:07:09.444655 20158 ts_manager.cc:194] Registered new tserver with Master: f8e0ab815ab44792821a8c03f997b5b0 (127.0.98.2:43323)
I20260924 04:07:09.445441 20158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43719
W20260924 04:07:09.550604 20480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:09.550901 20480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:09.550941 20480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:09.555238 20480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:09.555306 20480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:09.555372 20480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:07:09.560226 20480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:09.561270 20480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:09.562592 20480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:09.572422 20488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:09.572461 20485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:09.572432 20486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:09.572587 20480 server_base.cc:1034] running on GCE node
I20260924 04:07:09.573107 20480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:09.573814 20480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:09.575024 20480 hybrid_clock.cc:648] HybridClock initialized: now 1790222829574987 us; error 29 us; skew 500 ppm
I20260924 04:07:09.577163 20480 webserver.cc:460] Webserver started at http://127.0.98.3:35091/ using document root <none> and password file <none>
I20260924 04:07:09.577764 20480 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:09.577833 20480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:09.578035 20480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:09.580188 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4fe0bdf13374d348f1a82b19e36c7da"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "a9cb5e939dc3c4bd4c1f10854151c577"
server_key_iv: "c03df208e10e332d74d78ea7034fd838"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.580761 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4fe0bdf13374d348f1a82b19e36c7da"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "a9cb5e939dc3c4bd4c1f10854151c577"
server_key_iv: "c03df208e10e332d74d78ea7034fd838"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.584522 20480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:07:09.586987 20494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.588284 20480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:09.588406 20480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/wal
uuid: "e4fe0bdf13374d348f1a82b19e36c7da"
format_stamp: "Formatted at 2026-09-24 04:07:09 on dist-test-slave-zcsk"
server_key: "a9cb5e939dc3c4bd4c1f10854151c577"
server_key_iv: "c03df208e10e332d74d78ea7034fd838"
server_key_version: "encryptionkey@0"
I20260924 04:07:09.588508 20480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:09.608038 20480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:09.608736 20480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:09.608891 20480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:09.609467 20480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:07:09.610435 20480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:07:09.610482 20480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.610535 20480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:07:09.610564 20480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:09.619009 20480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42917
I20260924 04:07:09.619037 20607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42917 every 8 connection(s)
I20260924 04:07:09.619958 20480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:07:09.624013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 20480
I20260924 04:07:09.624148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:07:09.624442 392 external_mini_cluster.cc:1428] Setting key 83e174b9b7e9ee9766353aaf6b7bef5d
I20260924 04:07:09.629371 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.621756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39337 (local address 127.0.98.62:45571)
0924 04:07:09.621880 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.621884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.622969 (+ 1085us) server_negotiation.cc:407] Connection header received
0924 04:07:09.624637 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.624640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.624698 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.624776 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:09.625931 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.626569 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.627678 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.627919 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.627994 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.628997 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:09.628999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:09.629001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:09.629013 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:09.629030 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:09.629037 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:09.629088 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.629198 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.629247 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:07:09.630242 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.622146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45571 (local address 127.0.98.3:39337)
0924 04:07:09.622822 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.622866 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:09.624329 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:09.624996 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:09.625004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:09.625110 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:09.625774 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.625797 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.626683 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.626686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.627581 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.627590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.628030 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.628033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.628177 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.628749 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:07:09.628750 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:09.628794 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:07:09.628870 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:09.629097 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:09.629101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:09.629105 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:07:09.629192 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:07:09.629426 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260924 04:07:09.631302 20608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45571
I20260924 04:07:09.631538 20608 heartbeater.cc:464] Registering TS with master...
I20260924 04:07:09.632109 20608 heartbeater.cc:511] Master 127.0.98.62:45571 requested a full tablet report, sending...
I20260924 04:07:09.633212 20158 ts_manager.cc:194] Registered new tserver with Master: e4fe0bdf13374d348f1a82b19e36c7da (127.0.98.3:42917)
I20260924 04:07:09.633767 20158 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39337
I20260924 04:07:09.638163 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:07:09.643256 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.640030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.98.62:45571)
0924 04:07:09.640142 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.640146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.640313 (+ 167us) server_negotiation.cc:407] Connection header received
0924 04:07:09.640413 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.640415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.640456 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.640517 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:09.640741 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.641242 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.642000 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.642203 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.642269 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.642949 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:09.642952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:09.642954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:09.642964 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:09.642978 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:09.642984 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:09.643029 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.643101 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.643146 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:07:09.645962 20158 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:07:09.648952 20158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:07:09.664399 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.657983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39091 (local address 127.0.0.1:50220)
0924 04:07:09.659078 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.659090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:09.659237 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:09.659784 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:09.659786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:09.659799 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:09.659975 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.659981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.661253 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.661256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.662736 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.662747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.663788 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.663792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.663950 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.663998 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 04:07:09.664113 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:07:09.664240 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":149,"thread_start_us":97,"threads_started":1}
I20260924 04:07:09.664445 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.658525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.0.98.1:39091)
0924 04:07:09.658957 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.658969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.659166 (+ 197us) server_negotiation.cc:407] Connection header received
0924 04:07:09.659362 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.659366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.659508 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.659623 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:09.660078 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.661162 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.662871 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.663676 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.663754 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.663959 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.664155 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.664263 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":117,"threads_started":1}
I20260924 04:07:09.665028 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.657705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42917 (local address 127.0.0.1:44954)
0924 04:07:09.658667 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.658725 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:09.658860 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:09.659767 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:09.659770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:09.659788 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:09.659966 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.659973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.662456 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.662459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.663451 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.663462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.664650 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.664653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.664742 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.664766 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:07:09.664834 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:07:09.664899 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":19,"spinlock_wait_cycles":4480,"thread_start_us":69,"threads_started":1}
I20260924 04:07:09.665562 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.657881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.3:42917)
0924 04:07:09.659154 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.659165 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.659185 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:07:09.659256 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.659261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.659481 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.659623 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:09.661304 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.662345 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.663574 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.664298 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.664405 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.664587 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.665289 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.665403 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":151,"threads_started":1}
I20260924 04:07:09.665951 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.657773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43323 (local address 127.0.0.1:46174)
0924 04:07:09.658667 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.658724 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:09.658860 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:09.659634 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:09.659640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:09.659662 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:09.659790 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.659805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.661224 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.661228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.662736 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.662747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.665595 (+ 2848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.665597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.665691 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.665711 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:07:09.665775 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:07:09.665848 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":32,"spinlock_wait_cycles":6400,"thread_start_us":122,"threads_started":1}
I20260924 04:07:09.666460 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.658693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.0.98.2:43323)
0924 04:07:09.659140 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.659151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.659174 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:07:09.659254 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.659259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.659479 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.659623 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:09.659937 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.661120 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.664013 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.665274 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.665372 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.665541 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.666212 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.666314 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":114,"threads_started":1}
I20260924 04:07:09.667568 20280 tablet_service.cc:1467] Processing CreateTablet for tablet 615d274ff0df4022844eee6a0fd51669 (DEFAULT_TABLE table=test-table [id=562be30dc6c0451791944e7a3297b6bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:09.668085 20542 tablet_service.cc:1467] Processing CreateTablet for tablet 615d274ff0df4022844eee6a0fd51669 (DEFAULT_TABLE table=test-table [id=562be30dc6c0451791944e7a3297b6bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:09.668738 20280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615d274ff0df4022844eee6a0fd51669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:09.668972 20542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615d274ff0df4022844eee6a0fd51669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:09.669044 20411 tablet_service.cc:1467] Processing CreateTablet for tablet 615d274ff0df4022844eee6a0fd51669 (DEFAULT_TABLE table=test-table [id=562be30dc6c0451791944e7a3297b6bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:07:09.669948 20411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615d274ff0df4022844eee6a0fd51669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:09.674746 20625 tablet_bootstrap.cc:492] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Bootstrap starting.
I20260924 04:07:09.676182 20626 tablet_bootstrap.cc:492] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f: Bootstrap starting.
I20260924 04:07:09.676795 20627 tablet_bootstrap.cc:492] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: Bootstrap starting.
I20260924 04:07:09.677903 20625 tablet_bootstrap.cc:654] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:09.678121 20626 tablet_bootstrap.cc:654] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:09.678735 20626 log.cc:824] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:09.678730 20625 log.cc:824] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:09.678815 20627 tablet_bootstrap.cc:654] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:09.679543 20627 log.cc:824] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:09.680682 20626 tablet_bootstrap.cc:492] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f: No bootstrap required, opened a new log
I20260924 04:07:09.680678 20625 tablet_bootstrap.cc:492] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: No bootstrap required, opened a new log
I20260924 04:07:09.680868 20626 ts_tablet_manager.cc:1397] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:07:09.680868 20625 ts_tablet_manager.cc:1397] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:07:09.681118 20627 tablet_bootstrap.cc:492] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: No bootstrap required, opened a new log
I20260924 04:07:09.681303 20627 ts_tablet_manager.cc:1397] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:07:09.684700 20627 raft_consensus.cc:348] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.684924 20627 raft_consensus.cc:374] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:09.684976 20627 raft_consensus.cc:729] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e0ab815ab44792821a8c03f997b5b0, State: Initialized, Role: FOLLOWER
I20260924 04:07:09.685050 20626 raft_consensus.cc:348] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.685050 20625 raft_consensus.cc:348] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.685307 20626 raft_consensus.cc:374] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:09.685307 20625 raft_consensus.cc:374] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:09.685364 20625 raft_consensus.cc:729] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fe0bdf13374d348f1a82b19e36c7da, State: Initialized, Role: FOLLOWER
I20260924 04:07:09.685364 20626 raft_consensus.cc:729] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5489bb91bd38429dbcbfab2238b6135f, State: Initialized, Role: FOLLOWER
I20260924 04:07:09.685418 20627 consensus_queue.cc:260] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.685792 20626 consensus_queue.cc:260] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.685792 20625 consensus_queue.cc:260] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.686177 20477 heartbeater.cc:503] Master 127.0.98.62:45571 was elected leader, sending a full tablet report...
I20260924 04:07:09.686774 20626 ts_tablet_manager.cc:1428] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:07:09.686774 20625 ts_tablet_manager.cc:1428] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:07:09.687105 20346 heartbeater.cc:503] Master 127.0.98.62:45571 was elected leader, sending a full tablet report...
I20260924 04:07:09.687114 20608 heartbeater.cc:503] Master 127.0.98.62:45571 was elected leader, sending a full tablet report...
I20260924 04:07:09.688261 20627 ts_tablet_manager.cc:1428] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 04:07:09.754914 20347 tablet.cc:2369] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:07:09.820775 20631 raft_consensus.cc:482] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:07:09.821022 20631 raft_consensus.cc:504] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.822278 20631 leader_election.cc:288] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4fe0bdf13374d348f1a82b19e36c7da (127.0.98.3:42917), 5489bb91bd38429dbcbfab2238b6135f (127.0.98.1:39091)
I20260924 04:07:09.826479 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.822570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42917 (local address 127.0.98.2:57877)
0924 04:07:09.822784 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.822799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:09.822902 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:09.823307 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:09.823311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:09.823331 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:09.823453 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.823461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.824362 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.824365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.825266 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.825275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.826030 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.826034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.826131 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.826159 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:07:09.826226 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:07:09.826325 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":1}
I20260924 04:07:09.826511 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.822674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57877 (local address 127.0.98.3:42917)
0924 04:07:09.822812 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.822816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.822852 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:07:09.823071 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.823075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.823185 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.823285 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:09.823600 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.824253 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.825410 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.825908 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.825992 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.826048 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.826291 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.826369 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:07:09.826663 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.822789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37421 (local address 127.0.98.1:39091)
0924 04:07:09.822977 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.822982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.823253 (+ 271us) server_negotiation.cc:407] Connection header received
0924 04:07:09.823437 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.823440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.823492 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.823575 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:07:09.823935 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.824516 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.825640 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.826096 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.826193 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.826249 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.826463 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.826544 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 04:07:09.827031 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.822590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39091 (local address 127.0.98.2:37421)
0924 04:07:09.823144 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.823159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:09.823327 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:09.823640 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:09.823643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:09.823662 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:07:09.823790 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.823796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.824665 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.824667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.825518 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:09.825527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.826209 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.826212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:09.826312 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.826333 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:07:09.826806 (+ 473us) client_negotiation.cc:240] Negotiation successful
0924 04:07:09.826894 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
I20260924 04:07:09.827160 20562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d274ff0df4022844eee6a0fd51669" candidate_uuid: "f8e0ab815ab44792821a8c03f997b5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" is_pre_election: true
I20260924 04:07:09.827513 20562 raft_consensus.cc:2454] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e0ab815ab44792821a8c03f997b5b0 in term 0.
I20260924 04:07:09.827641 20300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d274ff0df4022844eee6a0fd51669" candidate_uuid: "f8e0ab815ab44792821a8c03f997b5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5489bb91bd38429dbcbfab2238b6135f" is_pre_election: true
I20260924 04:07:09.827924 20300 raft_consensus.cc:2454] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e0ab815ab44792821a8c03f997b5b0 in term 0.
I20260924 04:07:09.828001 20366 leader_election.cc:302] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4fe0bdf13374d348f1a82b19e36c7da, f8e0ab815ab44792821a8c03f997b5b0; no voters:
I20260924 04:07:09.828286 20631 raft_consensus.cc:2781] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:07:09.828368 20631 raft_consensus.cc:482] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:07:09.828400 20631 raft_consensus.cc:3037] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:09.829381 20631 raft_consensus.cc:504] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.829725 20631 leader_election.cc:288] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [CANDIDATE]: Term 1 election: Requested vote from peers e4fe0bdf13374d348f1a82b19e36c7da (127.0.98.3:42917), 5489bb91bd38429dbcbfab2238b6135f (127.0.98.1:39091)
I20260924 04:07:09.830060 20562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d274ff0df4022844eee6a0fd51669" candidate_uuid: "f8e0ab815ab44792821a8c03f997b5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fe0bdf13374d348f1a82b19e36c7da"
I20260924 04:07:09.830071 20300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d274ff0df4022844eee6a0fd51669" candidate_uuid: "f8e0ab815ab44792821a8c03f997b5b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5489bb91bd38429dbcbfab2238b6135f"
I20260924 04:07:09.830160 20562 raft_consensus.cc:3037] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:09.830180 20300 raft_consensus.cc:3037] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:09.831141 20562 raft_consensus.cc:2454] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e0ab815ab44792821a8c03f997b5b0 in term 1.
I20260924 04:07:09.831146 20300 raft_consensus.cc:2454] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e0ab815ab44792821a8c03f997b5b0 in term 1.
I20260924 04:07:09.831413 20366 leader_election.cc:302] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5489bb91bd38429dbcbfab2238b6135f, e4fe0bdf13374d348f1a82b19e36c7da, f8e0ab815ab44792821a8c03f997b5b0; no voters:
I20260924 04:07:09.831583 20631 raft_consensus.cc:2781] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:09.831818 20631 raft_consensus.cc:686] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 1 LEADER]: Becoming Leader. State: Replica: f8e0ab815ab44792821a8c03f997b5b0, State: Running, Role: LEADER
I20260924 04:07:09.832176 20631 consensus_queue.cc:237] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } }
I20260924 04:07:09.835263 20157 catalog_manager.cc:5495] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e0ab815ab44792821a8c03f997b5b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f8e0ab815ab44792821a8c03f997b5b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e0ab815ab44792821a8c03f997b5b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 } health_report { overall_health: UNKNOWN } } }
I20260924 04:07:09.856480 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.852996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.0.98.2:43323)
0924 04:07:09.853114 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:09.853118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:09.853130 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:07:09.853228 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:09.853230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:09.853305 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:09.853395 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:09.853607 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.854181 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.855008 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:09.855214 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:09.855299 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:09.855625 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:09.856099 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:09.856188 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:09.856282 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:07:09.856333 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:07:09.862682 20300 raft_consensus.cc:1261] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f [term 1 FOLLOWER]: Refusing update from remote peer f8e0ab815ab44792821a8c03f997b5b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:09.862682 20562 raft_consensus.cc:1261] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 1 FOLLOWER]: Refusing update from remote peer f8e0ab815ab44792821a8c03f997b5b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:09.863529 20635 consensus_queue.cc:1035] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fe0bdf13374d348f1a82b19e36c7da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:09.863744 20631 consensus_queue.cc:1035] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5489bb91bd38429dbcbfab2238b6135f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:07:09.870862 20609 tablet.cc:2369] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:07:09.871812 20640 mvcc.cc:205] Tried to move back new op lower bound from 7332752711109808128 to 7332752710995288064. Current Snapshot: MvccSnapshot[applied={T|T < 7332752711109808128}]
I20260924 04:07:09.880261 20158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:09.880445 20158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:07:09.882741 20158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feae1cbbcd694c37a570c191ef6a1881: Sending DeleteTablet for 3 replicas of tablet 615d274ff0df4022844eee6a0fd51669
I20260924 04:07:09.883544 20542 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d274ff0df4022844eee6a0fd51669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:09 UTC) from {username='slave'} at 127.0.0.1:44954
I20260924 04:07:09.883539 20411 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d274ff0df4022844eee6a0fd51669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:09 UTC) from {username='slave'} at 127.0.0.1:46174
I20260924 04:07:09.883844 20280 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d274ff0df4022844eee6a0fd51669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:07:09 UTC) from {username='slave'} at 127.0.0.1:50220
I20260924 04:07:09.884299 20649 tablet_replica.cc:330] T 615d274ff0df4022844eee6a0fd51669 P 5489bb91bd38429dbcbfab2238b6135f: stopping tablet replica
I20260924 04:07:09.884493 20647 tablet_replica.cc:330] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: stopping tablet replica
I20260924 04:07:09.884626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 20218
I20260924 04:07:09.885603 20647 raft_consensus.cc:2229] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:07:09.885946 20647 raft_consensus.cc:2258] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:09.886310 20648 tablet_replica.cc:330] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: stopping tablet replica
I20260924 04:07:09.886583 20648 raft_consensus.cc:2229] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:07:09.886938 20648 raft_consensus.cc:2258] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:07:09.887612 20647 ts_tablet_manager.cc:1905] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:09.889214 20648 ts_tablet_manager.cc:1905] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:07:09.890661 20647 ts_tablet_manager.cc:1918] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:07:09.890765 20647 log.cc:1196] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-2/wal/wals/615d274ff0df4022844eee6a0fd51669
I20260924 04:07:09.891069 20647 ts_tablet_manager.cc:1939] T 615d274ff0df4022844eee6a0fd51669 P e4fe0bdf13374d348f1a82b19e36c7da: Deleting consensus metadata
I20260924 04:07:09.892304 20648 ts_tablet_manager.cc:1918] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 04:07:09.892336 20143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39091 (error 108)
I20260924 04:07:09.892452 20144 catalog_manager.cc:4841] TS e4fe0bdf13374d348f1a82b19e36c7da (127.0.98.3:42917): tablet 615d274ff0df4022844eee6a0fd51669 (table test-table [id=562be30dc6c0451791944e7a3297b6bc]) successfully deleted
I20260924 04:07:09.892519 20648 log.cc:1196] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: Deleting WAL directory at /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222576786845-392-0/minicluster-data/ts-1/wal/wals/615d274ff0df4022844eee6a0fd51669
I20260924 04:07:09.893004 20648 ts_tablet_manager.cc:1939] T 615d274ff0df4022844eee6a0fd51669 P f8e0ab815ab44792821a8c03f997b5b0: Deleting consensus metadata
I20260924 04:07:09.893095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 20349
I20260924 04:07:09.895057 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.894791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39091 (local address 127.0.0.1:50230)
0924 04:07:09.894913 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.894980 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:07:09.895211 20143 catalog_manager.cc:4584] TS 5489bb91bd38429dbcbfab2238b6135f (127.0.98.1:39091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 615d274ff0df4022844eee6a0fd51669: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39091: connect: Connection refused (error 111)
I20260924 04:07:09.898379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 20480
I20260924 04:07:09.900657 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:09.898693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43323 (local address 127.0.0.1:46180)
0924 04:07:09.900505 (+ 1812us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:09.900572 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1699}
I20260924 04:07:09.904588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 20127
2026-09-24T04:07:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1131 ms)
[----------] 6 tests from EncryptionPolicyTest (10111 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:09 dist-test-slave-zcsk krb5kdc[20653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:09 dist-test-slave-zcsk krb5kdc[20653](info): set up 2 sockets
Sep 24 04:07:09 dist-test-slave-zcsk krb5kdc[20653](info): commencing operation
krb5kdc: starting...
W20260924 04:07:10.872025 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:10 dist-test-slave-zcsk krb5kdc[20653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:10.991098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:11.101879 20666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:11.102190 20666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:11.102253 20666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:11.106387 20666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:11.106455 20666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:11.106484 20666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:11.106508 20666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:11.106536 20666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:11.112021 20666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:11.113106 20666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:11.114133 20666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:11.122496 20671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:11.122534 20666 server_base.cc:1034] running on GCE node
W20260924 04:07:11.122480 20672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:11.122476 20674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:11.123447 20666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:11.124670 20666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:11.125836 20666 hybrid_clock.cc:648] HybridClock initialized: now 1790222831125814 us; error 34 us; skew 500 ppm
Sep 24 04:07:11 dist-test-slave-zcsk krb5kdc[20653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:11.128634 20666 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:11.129570 20666 webserver.cc:460] Webserver started at http://127.0.0.1:45577/ using document root <none> and password file <none>
I20260924 04:07:11.130105 20666 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:11.130172 20666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:11.130367 20666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:11.132459 20666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "e910f3e1fd864a7683ac5054999b791f"
format_stamp: "Formatted at 2026-09-24 04:07:11 on dist-test-slave-zcsk"
server_key: "368d6389036edfc35c0394813b49a06f"
server_key_iv: "05ef068e2930ffc5bb72ae31e6908af6"
server_key_version: "encryptionkey@0"
I20260924 04:07:11.133011 20666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "e910f3e1fd864a7683ac5054999b791f"
format_stamp: "Formatted at 2026-09-24 04:07:11 on dist-test-slave-zcsk"
server_key: "368d6389036edfc35c0394813b49a06f"
server_key_iv: "05ef068e2930ffc5bb72ae31e6908af6"
server_key_version: "encryptionkey@0"
I20260924 04:07:11.136804 20666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:07:11.139240 20681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:11.140542 20666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:11.140686 20666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "e910f3e1fd864a7683ac5054999b791f"
format_stamp: "Formatted at 2026-09-24 04:07:11 on dist-test-slave-zcsk"
server_key: "368d6389036edfc35c0394813b49a06f"
server_key_iv: "05ef068e2930ffc5bb72ae31e6908af6"
server_key_version: "encryptionkey@0"
I20260924 04:07:11.140787 20666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:11.173126 20666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:11.176441 20666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:11.176640 20666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:11.183495 20666 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41533
I20260924 04:07:11.183506 20733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41533 every 8 connection(s)
I20260924 04:07:11.184480 20666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:11.187191 20734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:11.187556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 20666
I20260924 04:07:11.187674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:11.187956 392 external_mini_cluster.cc:1428] Setting key 1ca749a32944f5e97629beab11638a45
I20260924 04:07:11.194412 20734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f: Bootstrap starting.
I20260924 04:07:11.196789 20734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f: Neither blocks nor log segments found. Creating new log.
Sep 24 04:07:11 dist-test-slave-zcsk krb5kdc[20653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:07:11.197681 20734 log.cc:824] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:11.200294 20734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f: No bootstrap required, opened a new log
I20260924 04:07:11.203392 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:11.189353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34597 (local address 127.0.0.1:41533)
0924 04:07:11.189759 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:11.189779 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:11.189821 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:07:11.190565 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:11.190589 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:11.191069 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:11.191457 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:11.192239 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:11.193750 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:11.194628 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:11.195105 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:11.195265 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:11.197975 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:11.197981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:11.198000 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:11.198033 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:11.200044 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:11.200574 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:11.200580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:11.200586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:11.200738 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:11.201020 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:11.201024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:11.201026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:11.201539 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:11.201743 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:11.202061 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 04:07:11.202305 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260924 04:07:11.204859 20734 raft_consensus.cc:348] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910f3e1fd864a7683ac5054999b791f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260924 04:07:11.205081 20734 raft_consensus.cc:374] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:11.205113 20734 raft_consensus.cc:729] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910f3e1fd864a7683ac5054999b791f, State: Initialized, Role: FOLLOWER
I20260924 04:07:11.205588 20734 consensus_queue.cc:260] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910f3e1fd864a7683ac5054999b791f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260924 04:07:11.205747 20734 raft_consensus.cc:388] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:11.205812 20734 raft_consensus.cc:482] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:11.205909 20734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:11.206595 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:11.206962 20734 raft_consensus.cc:504] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910f3e1fd864a7683ac5054999b791f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260924 04:07:11.207288 20734 leader_election.cc:302] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e910f3e1fd864a7683ac5054999b791f; no voters:
I20260924 04:07:11.207582 20734 leader_election.cc:288] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:11.207698 20739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:11.207952 20739 raft_consensus.cc:686] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [term 1 LEADER]: Becoming Leader. State: Replica: e910f3e1fd864a7683ac5054999b791f, State: Running, Role: LEADER
I20260924 04:07:11.208325 20739 consensus_queue.cc:237] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910f3e1fd864a7683ac5054999b791f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260924 04:07:11.209013 20734 sys_catalog.cc:564] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:11.210456 20745 sys_catalog.cc:455] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e910f3e1fd864a7683ac5054999b791f. Latest consensus state: current_term: 1 leader_uuid: "e910f3e1fd864a7683ac5054999b791f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910f3e1fd864a7683ac5054999b791f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } } }
I20260924 04:07:11.210584 20745 sys_catalog.cc:458] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:11.210511 20744 sys_catalog.cc:455] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e910f3e1fd864a7683ac5054999b791f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e910f3e1fd864a7683ac5054999b791f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } } }
I20260924 04:07:11.210649 20744 sys_catalog.cc:458] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:11.210951 20757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:11.214299 20757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:11.217152 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:11.208880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53671 (local address 127.0.0.1:41533)
0924 04:07:11.208987 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:11.208991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:11.209006 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:07:11.209043 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:11.209046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:11.209092 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:11.209214 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:11.209824 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:11.210608 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:11.211680 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:11.211962 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:11.212060 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:11.213544 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:11.213547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:11.213550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:11.213575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:11.215189 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:11.215808 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:11.215812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:11.215813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:11.215934 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:11.216299 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:11.216302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:11.216304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:11.216490 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:11.216624 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:11.216899 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 04:07:11.216989 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 04:07:11.220124 20757 catalog_manager.cc:1347] Generated new cluster ID: 3fa294eaf4dc4716ab744ecc296b5ca2
I20260924 04:07:11.220202 20757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:11.245119 20757 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:11.246127 20757 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:11.252447 20757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e910f3e1fd864a7683ac5054999b791f: Generated new TSK 0
I20260924 04:07:11.253263 20757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:11.264377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 20666
W20260924 04:07:11.269279 20651 connection.cc:535] client connection to 127.0.0.1:41533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41533): Connection reset by peer (error 104)
2026-09-24T04:07:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:11 dist-test-slave-zcsk krb5kdc[20765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:11 dist-test-slave-zcsk krb5kdc[20765](info): set up 2 sockets
Sep 24 04:07:11 dist-test-slave-zcsk krb5kdc[20765](info): commencing operation
krb5kdc: starting...
W20260924 04:07:12.223747 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:12 dist-test-slave-zcsk krb5kdc[20765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:12.342866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:12.454416 20778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:12.454707 20778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:12.454754 20778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:12.458873 20778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:12.458954 20778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:12.458976 20778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:12.458995 20778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:12.459012 20778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:12.464623 20778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:12.465736 20778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:12.466800 20778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:12.475333 20783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:12.475337 20784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:12.475337 20786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:12.475391 20778 server_base.cc:1034] running on GCE node
I20260924 04:07:12.476195 20778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:12.477380 20778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:12.478552 20778 hybrid_clock.cc:648] HybridClock initialized: now 1790222832478524 us; error 40 us; skew 500 ppm
Sep 24 04:07:12 dist-test-slave-zcsk krb5kdc[20765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:12.481426 20778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:12.482352 20778 webserver.cc:460] Webserver started at http://127.0.0.1:33163/ using document root <none> and password file <none>
I20260924 04:07:12.482913 20778 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:12.482985 20778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:12.483194 20778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:12.485353 20778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "d19db8f9417b4091814e9adce89fc225"
format_stamp: "Formatted at 2026-09-24 04:07:12 on dist-test-slave-zcsk"
server_key: "63091c26fa3f1925541098aebdc89833"
server_key_iv: "0b4408678e24f4bc17dfd46ae3137e21"
server_key_version: "encryptionkey@0"
I20260924 04:07:12.485889 20778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "d19db8f9417b4091814e9adce89fc225"
format_stamp: "Formatted at 2026-09-24 04:07:12 on dist-test-slave-zcsk"
server_key: "63091c26fa3f1925541098aebdc89833"
server_key_iv: "0b4408678e24f4bc17dfd46ae3137e21"
server_key_version: "encryptionkey@0"
I20260924 04:07:12.489717 20778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:07:12.492293 20793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:12.493587 20778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:12.493721 20778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "d19db8f9417b4091814e9adce89fc225"
format_stamp: "Formatted at 2026-09-24 04:07:12 on dist-test-slave-zcsk"
server_key: "63091c26fa3f1925541098aebdc89833"
server_key_iv: "0b4408678e24f4bc17dfd46ae3137e21"
server_key_version: "encryptionkey@0"
I20260924 04:07:12.493860 20778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:12.514564 20778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:12.518107 20778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:12.518298 20778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:12.525514 20778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45295
I20260924 04:07:12.525573 20845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45295 every 8 connection(s)
I20260924 04:07:12.526551 20778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:12.529127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 20778
I20260924 04:07:12.529273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:12.529582 392 external_mini_cluster.cc:1428] Setting key 4923360cd015330f7e3ab28497e2b219
I20260924 04:07:12.529894 20846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:12.536499 20846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225: Bootstrap starting.
I20260924 04:07:12.538812 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:12.531045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35127 (local address 127.0.0.1:45295)
0924 04:07:12.531533 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:12.531549 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:12.531592 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:07:12.532703 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:12.532724 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:12.533087 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:12.533399 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:12.533805 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:12.534780 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:12.535647 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:12.536173 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:12.536317 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:12.537190 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:12.537194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:12.537207 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:12.537596 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:12.537651 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:12.537673 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:12.537781 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:12.537952 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:07:12.538088 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":152,"threads_started":1}
I20260924 04:07:12.538812 20846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:12.539592 20846 log.cc:824] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:12.541675 20846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225: No bootstrap required, opened a new log
I20260924 04:07:12.541783 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:12.546108 20846 raft_consensus.cc:348] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19db8f9417b4091814e9adce89fc225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } }
I20260924 04:07:12.546438 20846 raft_consensus.cc:374] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:12.546501 20846 raft_consensus.cc:729] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19db8f9417b4091814e9adce89fc225, State: Initialized, Role: FOLLOWER
I20260924 04:07:12.547019 20846 consensus_queue.cc:260] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19db8f9417b4091814e9adce89fc225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } }
I20260924 04:07:12.547197 20846 raft_consensus.cc:388] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:12.547272 20846 raft_consensus.cc:482] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:12.547374 20846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:12.548066 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:12.543967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49569 (local address 127.0.0.1:45295)
0924 04:07:12.544072 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:12.544076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:12.544270 (+ 194us) server_negotiation.cc:407] Connection header received
0924 04:07:12.544362 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:12.544364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:12.544403 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:12.544522 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:12.544807 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:12.545298 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:12.546370 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:12.546681 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:12.546767 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:12.547640 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:12.547643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:12.547645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:12.547656 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:12.547680 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:12.547689 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:12.547753 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:12.547882 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 04:07:12.547936 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:07:12.548602 20846 raft_consensus.cc:504] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19db8f9417b4091814e9adce89fc225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } }
I20260924 04:07:12.548974 20846 leader_election.cc:302] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d19db8f9417b4091814e9adce89fc225; no voters:
I20260924 04:07:12.549301 20846 leader_election.cc:288] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:12.549399 20856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:12.549610 20856 raft_consensus.cc:686] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [term 1 LEADER]: Becoming Leader. State: Replica: d19db8f9417b4091814e9adce89fc225, State: Running, Role: LEADER
I20260924 04:07:12.549933 20856 consensus_queue.cc:237] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19db8f9417b4091814e9adce89fc225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } }
I20260924 04:07:12.550571 20846 sys_catalog.cc:564] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:12.551689 20857 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d19db8f9417b4091814e9adce89fc225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19db8f9417b4091814e9adce89fc225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } } }
I20260924 04:07:12.551822 20857 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:12.552069 20858 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d19db8f9417b4091814e9adce89fc225. Latest consensus state: current_term: 1 leader_uuid: "d19db8f9417b4091814e9adce89fc225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19db8f9417b4091814e9adce89fc225" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } } }
I20260924 04:07:12.552161 20858 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225 [sys.catalog]: This master's current role is: LEADER
W20260924 04:07:12.555292 20871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:12.555364 20871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:12.555706 20865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:12.556476 20865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:12.562199 20865 catalog_manager.cc:1347] Generated new cluster ID: 67af5288eced46158d780e6680f02177
I20260924 04:07:12.562286 20865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:12.571425 20865 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:12.572503 20865 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:12.580606 20865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d19db8f9417b4091814e9adce89fc225: Generated new TSK 0
I20260924 04:07:12.581378 20865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:12.593714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 20778
W20260924 04:07:12.599121 20763 connection.cc:535] client connection to 127.0.0.1:45295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45295): Connection reset by peer (error 104)
2026-09-24T04:07:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:12 dist-test-slave-zcsk krb5kdc[20877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:12 dist-test-slave-zcsk krb5kdc[20877](info): set up 2 sockets
Sep 24 04:07:12 dist-test-slave-zcsk krb5kdc[20877](info): commencing operation
krb5kdc: starting...
W20260924 04:07:13.547955 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:13 dist-test-slave-zcsk krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:13.668838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:13.778945 20890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:13.779239 20890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:13.779281 20890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:13.783363 20890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:13.783448 20890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:13.783468 20890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:13.783483 20890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:13.783497 20890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:13.788933 20890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:13.789985 20890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:13.790994 20890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:13.799576 20896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:13.799616 20895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:13.799616 20898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:13.799860 20890 server_base.cc:1034] running on GCE node
I20260924 04:07:13.800448 20890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:13.801666 20890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:13.802853 20890 hybrid_clock.cc:648] HybridClock initialized: now 1790222833802826 us; error 44 us; skew 500 ppm
Sep 24 04:07:13 dist-test-slave-zcsk krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:13.805645 20890 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:13.806596 20890 webserver.cc:460] Webserver started at http://127.0.0.1:37535/ using document root <none> and password file <none>
I20260924 04:07:13.807186 20890 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:13.807258 20890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:13.807457 20890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:13.809600 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "fdbd610e5db549459e1e76d1fc9c6ef5"
format_stamp: "Formatted at 2026-09-24 04:07:13 on dist-test-slave-zcsk"
server_key: "8032b64d6e2648a5720da0b9c5f10435"
server_key_iv: "d543857203b972fd3acbc30bef023563"
server_key_version: "encryptionkey@0"
I20260924 04:07:13.810149 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdbd610e5db549459e1e76d1fc9c6ef5"
format_stamp: "Formatted at 2026-09-24 04:07:13 on dist-test-slave-zcsk"
server_key: "8032b64d6e2648a5720da0b9c5f10435"
server_key_iv: "d543857203b972fd3acbc30bef023563"
server_key_version: "encryptionkey@0"
I20260924 04:07:13.814059 20890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:07:13.816664 20905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:13.818034 20890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:13.818147 20890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "fdbd610e5db549459e1e76d1fc9c6ef5"
format_stamp: "Formatted at 2026-09-24 04:07:13 on dist-test-slave-zcsk"
server_key: "8032b64d6e2648a5720da0b9c5f10435"
server_key_iv: "d543857203b972fd3acbc30bef023563"
server_key_version: "encryptionkey@0"
I20260924 04:07:13.818245 20890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:13.822935 20890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:13.825958 20890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:13.826114 20890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:13.832757 20890 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39467
I20260924 04:07:13.832769 20957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39467 every 8 connection(s)
I20260924 04:07:13.833715 20890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:13.834774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 20890
I20260924 04:07:13.834920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:13.835220 392 external_mini_cluster.cc:1428] Setting key aa189c67440c628f58278a93efdb2e1f
I20260924 04:07:13.837128 20958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:13.840363 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:13.836635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37301 (local address 127.0.0.1:39467)
0924 04:07:13.837050 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:13.837066 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:13.837100 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:07:13.837774 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:13.837803 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:13.838113 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:13.838424 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:13.839165 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:13.839169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:13.839190 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:13.839200 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:13.839252 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:13.839271 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:13.839391 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:13.839462 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:07:13.839609 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":138,"threads_started":1}
I20260924 04:07:13.842940 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:13.842940 20958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5: Bootstrap starting.
I20260924 04:07:13.845268 20958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:13.846058 20958 log.cc:824] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:13.846477 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:13.844821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40371 (local address 127.0.0.1:39467)
0924 04:07:13.844925 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:13.844929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:13.845091 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:07:13.845214 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:13.845216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:13.845245 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:13.845301 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:13.846000 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:13.846004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:13.846007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:13.846018 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:13.846055 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:13.846066 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:13.846137 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:13.846334 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 04:07:13.846364 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:07:13.848138 20958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5: No bootstrap required, opened a new log
I20260924 04:07:13.851797 20958 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbd610e5db549459e1e76d1fc9c6ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } }
I20260924 04:07:13.852104 20958 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:13.852156 20958 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbd610e5db549459e1e76d1fc9c6ef5, State: Initialized, Role: FOLLOWER
I20260924 04:07:13.852543 20958 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbd610e5db549459e1e76d1fc9c6ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } }
I20260924 04:07:13.852653 20958 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:13.852696 20958 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:13.852767 20958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:13.853641 20958 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbd610e5db549459e1e76d1fc9c6ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } }
I20260924 04:07:13.853963 20958 leader_election.cc:302] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdbd610e5db549459e1e76d1fc9c6ef5; no voters:
I20260924 04:07:13.854244 20958 leader_election.cc:288] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:13.854358 20968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:13.854578 20968 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [term 1 LEADER]: Becoming Leader. State: Replica: fdbd610e5db549459e1e76d1fc9c6ef5, State: Running, Role: LEADER
I20260924 04:07:13.854966 20968 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbd610e5db549459e1e76d1fc9c6ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } }
I20260924 04:07:13.855535 20958 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:13.856650 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdbd610e5db549459e1e76d1fc9c6ef5. Latest consensus state: current_term: 1 leader_uuid: "fdbd610e5db549459e1e76d1fc9c6ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbd610e5db549459e1e76d1fc9c6ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } } }
I20260924 04:07:13.856783 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:13.857038 20969 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdbd610e5db549459e1e76d1fc9c6ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbd610e5db549459e1e76d1fc9c6ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39467 } } }
I20260924 04:07:13.857125 20969 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:13.857339 20977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:13.860553 20977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:13.866572 20977 catalog_manager.cc:1347] Generated new cluster ID: 3635993cfffa4011be63791ca8a39316
I20260924 04:07:13.866664 20977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:13.879843 20977 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:13.880204 20977 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:13.890214 20977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdbd610e5db549459e1e76d1fc9c6ef5: Generated new TSK 0
I20260924 04:07:13.891119 20977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:13.904874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 20890
2026-09-24T04:07:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:13 dist-test-slave-zcsk krb5kdc[20989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:13 dist-test-slave-zcsk krb5kdc[20989](info): set up 2 sockets
Sep 24 04:07:13 dist-test-slave-zcsk krb5kdc[20989](info): commencing operation
krb5kdc: starting...
W20260924 04:07:14.863801 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:14 dist-test-slave-zcsk krb5kdc[20989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:14.983639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:15.093839 21002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:15.094406 21002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:15.094475 21002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:15.098631 21002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:15.098702 21002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:15.098729 21002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:15.098760 21002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:15.098793 21002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:15.104212 21002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:15.105289 21002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:15.106304 21002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:15.115242 21008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:15.115370 21007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:15.115494 21002 server_base.cc:1034] running on GCE node
W20260924 04:07:15.115357 21010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:15.116410 21002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:15.117558 21002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:15.118743 21002 hybrid_clock.cc:648] HybridClock initialized: now 1790222835118725 us; error 32 us; skew 500 ppm
Sep 24 04:07:15 dist-test-slave-zcsk krb5kdc[20989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:15.125528 21002 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:15.126585 21002 webserver.cc:460] Webserver started at http://127.0.0.1:37969/ using document root <none> and password file <none>
I20260924 04:07:15.127171 21002 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:15.127241 21002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:15.127458 21002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:15.129619 21002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "ed58cc4a6ec44555b66de206a42aa380"
format_stamp: "Formatted at 2026-09-24 04:07:15 on dist-test-slave-zcsk"
server_key: "398c1d8840955a0305d9d0a3c9855907"
server_key_iv: "3867282fc5fe707659e0a111fac6889e"
server_key_version: "encryptionkey@0"
I20260924 04:07:15.130192 21002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed58cc4a6ec44555b66de206a42aa380"
format_stamp: "Formatted at 2026-09-24 04:07:15 on dist-test-slave-zcsk"
server_key: "398c1d8840955a0305d9d0a3c9855907"
server_key_iv: "3867282fc5fe707659e0a111fac6889e"
server_key_version: "encryptionkey@0"
I20260924 04:07:15.134115 21002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:07:15.136756 21017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:15.138231 21002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:15.138356 21002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "ed58cc4a6ec44555b66de206a42aa380"
format_stamp: "Formatted at 2026-09-24 04:07:15 on dist-test-slave-zcsk"
server_key: "398c1d8840955a0305d9d0a3c9855907"
server_key_iv: "3867282fc5fe707659e0a111fac6889e"
server_key_version: "encryptionkey@0"
I20260924 04:07:15.138463 21002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:15.152621 21002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:15.155653 21002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:15.155846 21002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:15.162848 21002 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41161
I20260924 04:07:15.162875 21069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41161 every 8 connection(s)
I20260924 04:07:15.163805 21002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:15.166585 21070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:15.170120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21002
I20260924 04:07:15.170282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:15.170572 392 external_mini_cluster.cc:1428] Setting key 13a637a26abf70292ff3fa89e3af732d
I20260924 04:07:15.172868 21070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380: Bootstrap starting.
I20260924 04:07:15.175139 21070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:15.175931 21070 log.cc:824] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:15.177918 21070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380: No bootstrap required, opened a new log
Sep 24 04:07:15 dist-test-slave-zcsk krb5kdc[20989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:07:15.182036 21070 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed58cc4a6ec44555b66de206a42aa380" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } }
I20260924 04:07:15.182292 21070 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:15.182353 21070 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed58cc4a6ec44555b66de206a42aa380, State: Initialized, Role: FOLLOWER
I20260924 04:07:15.182816 21070 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed58cc4a6ec44555b66de206a42aa380" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } }
I20260924 04:07:15.182953 21070 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:15.183020 21070 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:15.183120 21070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:15.184073 21070 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed58cc4a6ec44555b66de206a42aa380" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } }
I20260924 04:07:15.184419 21070 leader_election.cc:302] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed58cc4a6ec44555b66de206a42aa380; no voters:
I20260924 04:07:15.184737 21070 leader_election.cc:288] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:15.185241 21076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:15.185684 21076 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [term 1 LEADER]: Becoming Leader. State: Replica: ed58cc4a6ec44555b66de206a42aa380, State: Running, Role: LEADER
I20260924 04:07:15.185689 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:15.172101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39095 (local address 127.0.0.1:41161)
0924 04:07:15.172571 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:15.172586 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:15.172626 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:07:15.173358 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:15.173377 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:15.173749 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:15.174068 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:15.174859 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:15.175975 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:15.176999 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:15.177482 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:15.177624 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:15.180318 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:15.180323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:15.180338 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:15.180367 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:15.182157 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:15.182660 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:15.182666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:15.182671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:15.182790 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:15.183124 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:15.183127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:15.183128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:15.183571 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:15.183774 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:15.184107 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 04:07:15.184524 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":144,"threads_started":1}
I20260924 04:07:15.186107 21076 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed58cc4a6ec44555b66de206a42aa380" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } }
I20260924 04:07:15.186174 21070 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:15.188673 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed58cc4a6ec44555b66de206a42aa380. Latest consensus state: current_term: 1 leader_uuid: "ed58cc4a6ec44555b66de206a42aa380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed58cc4a6ec44555b66de206a42aa380" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } } }
I20260924 04:07:15.188832 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:15.188874 21077 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed58cc4a6ec44555b66de206a42aa380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed58cc4a6ec44555b66de206a42aa380" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41161 } } }
I20260924 04:07:15.188964 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:15.189044 21077 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380 [sys.catalog]: This master's current role is: LEADER
W20260924 04:07:15.191727 21089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:15.191792 21089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:15.191866 21090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:15.192708 21090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:15.198428 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:15.190794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49993 (local address 127.0.0.1:41161)
0924 04:07:15.190899 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:15.190903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:15.191060 (+ 157us) server_negotiation.cc:407] Connection header received
0924 04:07:15.191196 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:15.191199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:15.191247 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:15.191319 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:15.192099 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:15.192616 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:15.193431 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:15.193671 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:15.193755 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:15.195189 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:15.195193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:15.195196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:15.195222 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:15.196882 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:15.197319 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:15.197323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:15.197325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:15.197406 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:15.197640 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:15.197643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:15.197645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:15.197821 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:15.197932 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:15.198163 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:07:15.198274 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:07:15.198748 21090 catalog_manager.cc:1347] Generated new cluster ID: d4c11c83c0474164bc263152f516ecaa
I20260924 04:07:15.198807 21090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:15.213994 21090 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:15.214978 21090 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:15.221269 21090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed58cc4a6ec44555b66de206a42aa380: Generated new TSK 0
I20260924 04:07:15.222000 21090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:15.226822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21002
2026-09-24T04:07:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:15 dist-test-slave-zcsk krb5kdc[21101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:15 dist-test-slave-zcsk krb5kdc[21101](info): set up 2 sockets
Sep 24 04:07:15 dist-test-slave-zcsk krb5kdc[21101](info): commencing operation
krb5kdc: starting...
W20260924 04:07:16.195947 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:16 dist-test-slave-zcsk krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:16.315297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:16.423807 21114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:16.424121 21114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:16.424183 21114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:16.428164 21114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:16.428224 21114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:16.428251 21114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:16.428272 21114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:16.428299 21114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:16.433552 21114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:16.434636 21114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:16.435658 21114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:16.444686 21114 server_base.cc:1034] running on GCE node
W20260924 04:07:16.444610 21119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:16.444629 21120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:16.444618 21122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:16.445493 21114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:16.446659 21114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:16.447849 21114 hybrid_clock.cc:648] HybridClock initialized: now 1790222836447794 us; error 68 us; skew 500 ppm
Sep 24 04:07:16 dist-test-slave-zcsk krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:16.450848 21114 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:16.451826 21114 webserver.cc:460] Webserver started at http://127.0.0.1:38861/ using document root <none> and password file <none>
I20260924 04:07:16.452405 21114 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:16.452476 21114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:16.452702 21114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:16.454803 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "6eb8bc528e6f41b8a00da3cb86727351"
format_stamp: "Formatted at 2026-09-24 04:07:16 on dist-test-slave-zcsk"
server_key: "6d8b9eafe7ca3a6d60fd1f03217e780c"
server_key_iv: "37c8065e00dd0ab285e65f919d5b20c9"
server_key_version: "encryptionkey@0"
I20260924 04:07:16.455348 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "6eb8bc528e6f41b8a00da3cb86727351"
format_stamp: "Formatted at 2026-09-24 04:07:16 on dist-test-slave-zcsk"
server_key: "6d8b9eafe7ca3a6d60fd1f03217e780c"
server_key_iv: "37c8065e00dd0ab285e65f919d5b20c9"
server_key_version: "encryptionkey@0"
I20260924 04:07:16.459161 21114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 04:07:16.461728 21129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:16.463073 21114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:07:16.463202 21114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "6eb8bc528e6f41b8a00da3cb86727351"
format_stamp: "Formatted at 2026-09-24 04:07:16 on dist-test-slave-zcsk"
server_key: "6d8b9eafe7ca3a6d60fd1f03217e780c"
server_key_iv: "37c8065e00dd0ab285e65f919d5b20c9"
server_key_version: "encryptionkey@0"
I20260924 04:07:16.463310 21114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:16.481601 21114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:16.484918 21114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:16.485121 21114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:16.492131 21114 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35243
I20260924 04:07:16.492172 21181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35243 every 8 connection(s)
I20260924 04:07:16.493127 21114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:16.496150 21182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:16.501423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21114
I20260924 04:07:16.501571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:16.501835 392 external_mini_cluster.cc:1428] Setting key 47a1b485cde010474ad735290b545226
I20260924 04:07:16.502279 21182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351: Bootstrap starting.
I20260924 04:07:16.505015 21182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:16.505776 21182 log.cc:824] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:16.507704 21182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351: No bootstrap required, opened a new log
I20260924 04:07:16.511575 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:16.503651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37643 (local address 127.0.0.1:35243)
0924 04:07:16.504179 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:16.504195 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:16.504232 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:07:16.504929 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:16.504947 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:16.505360 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:16.505699 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:16.506133 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:16.507119 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:16.508076 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:16.508575 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:16.508732 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:16.509661 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:16.509666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:16.509681 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:16.510166 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:16.510225 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:16.510251 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:16.510345 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:16.510495 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 04:07:16.510728 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":167,"threads_started":1}
I20260924 04:07:16.511684 21182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8bc528e6f41b8a00da3cb86727351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 04:07:16.512017 21182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:16.512079 21182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb8bc528e6f41b8a00da3cb86727351, State: Initialized, Role: FOLLOWER
I20260924 04:07:16.512508 21182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8bc528e6f41b8a00da3cb86727351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 04:07:16.512645 21182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:16.512699 21182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:16.512786 21182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:16.513990 21182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8bc528e6f41b8a00da3cb86727351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 04:07:16.514374 21182 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb8bc528e6f41b8a00da3cb86727351; no voters:
I20260924 04:07:16.514735 21182 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:16.514832 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:16.514894 21187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:16.515084 21187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [term 1 LEADER]: Becoming Leader. State: Replica: 6eb8bc528e6f41b8a00da3cb86727351, State: Running, Role: LEADER
I20260924 04:07:16.515458 21187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8bc528e6f41b8a00da3cb86727351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } }
I20260924 04:07:16.516191 21182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:16.517233 21189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eb8bc528e6f41b8a00da3cb86727351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8bc528e6f41b8a00da3cb86727351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } } }
I20260924 04:07:16.517369 21189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:16.517632 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb8bc528e6f41b8a00da3cb86727351. Latest consensus state: current_term: 1 leader_uuid: "6eb8bc528e6f41b8a00da3cb86727351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb8bc528e6f41b8a00da3cb86727351" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35243 } } }
I20260924 04:07:16.517765 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:16.517936 21201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:16.521560 21201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:16.523023 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:16.516947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54413 (local address 127.0.0.1:35243)
0924 04:07:16.517054 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:16.517059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:16.518459 (+ 1400us) server_negotiation.cc:407] Connection header received
0924 04:07:16.518863 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:16.518866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:16.518920 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:16.518992 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:16.519267 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:16.520011 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:16.521134 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:16.521460 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:16.521572 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:16.522550 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:16.522554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:16.522557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:16.522572 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:16.522602 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:16.522616 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:16.522689 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:16.522822 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:07:16.522882 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:07:16.528265 21201 catalog_manager.cc:1347] Generated new cluster ID: 7977907e80bd422395bd64f71de6cfce
I20260924 04:07:16.528422 21201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:16.541677 21201 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:16.542742 21201 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:16.549465 21201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eb8bc528e6f41b8a00da3cb86727351: Generated new TSK 0
I20260924 04:07:16.550309 21201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:16.554865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21114
2026-09-24T04:07:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:16 dist-test-slave-zcsk krb5kdc[21213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:16 dist-test-slave-zcsk krb5kdc[21213](info): set up 2 sockets
Sep 24 04:07:16 dist-test-slave-zcsk krb5kdc[21213](info): commencing operation
krb5kdc: starting...
W20260924 04:07:17.507898 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:17 dist-test-slave-zcsk krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:17.631119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:17.741201 21226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:17.741505 21226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:17.741568 21226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:17.745683 21226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:17.745764 21226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:17.745791 21226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:17.745813 21226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:17.745837 21226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:17.751389 21226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:17.752607 21226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:17.753641 21226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:17.762265 21231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:17.762341 21234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:17.762286 21232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:17.762570 21226 server_base.cc:1034] running on GCE node
I20260924 04:07:17.763180 21226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:17.764464 21226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:17.765642 21226 hybrid_clock.cc:648] HybridClock initialized: now 1790222837765612 us; error 45 us; skew 500 ppm
Sep 24 04:07:17 dist-test-slave-zcsk krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:17.768699 21226 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:17.769657 21226 webserver.cc:460] Webserver started at http://127.0.0.1:44637/ using document root <none> and password file <none>
I20260924 04:07:17.770215 21226 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:17.770282 21226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:17.770476 21226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:17.772590 21226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c"
format_stamp: "Formatted at 2026-09-24 04:07:17 on dist-test-slave-zcsk"
server_key: "761b9ab4a490849d2d01a8617a270a30"
server_key_iv: "a2fc28a222fb45b090a25a9cf4923de8"
server_key_version: "encryptionkey@0"
I20260924 04:07:17.773131 21226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c"
format_stamp: "Formatted at 2026-09-24 04:07:17 on dist-test-slave-zcsk"
server_key: "761b9ab4a490849d2d01a8617a270a30"
server_key_iv: "a2fc28a222fb45b090a25a9cf4923de8"
server_key_version: "encryptionkey@0"
I20260924 04:07:17.776755 21226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:07:17.779171 21241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:17.780440 21226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:17.780556 21226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c"
format_stamp: "Formatted at 2026-09-24 04:07:17 on dist-test-slave-zcsk"
server_key: "761b9ab4a490849d2d01a8617a270a30"
server_key_iv: "a2fc28a222fb45b090a25a9cf4923de8"
server_key_version: "encryptionkey@0"
I20260924 04:07:17.780656 21226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:17.785722 21226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:17.788800 21226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:17.788981 21226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:17.795591 21226 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33351
I20260924 04:07:17.795624 21293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33351 every 8 connection(s)
I20260924 04:07:17.796589 21226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:17.797129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21226
I20260924 04:07:17.797276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:17.797575 392 external_mini_cluster.cc:1428] Setting key 5c31b09e8ebaaeb7072b824b500d201a
I20260924 04:07:17.800009 21294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:17.802757 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:17.798960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35863 (local address 127.0.0.1:33351)
0924 04:07:17.799398 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:17.799413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:17.799449 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:07:17.800203 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:17.800223 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:17.800523 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:17.800842 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:17.801596 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:17.801601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:17.801614 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:17.801624 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:17.801678 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:17.801697 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:17.801823 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:17.801901 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 04:07:17.802058 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":127,"threads_started":1}
I20260924 04:07:17.805483 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:17.805938 21294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c: Bootstrap starting.
I20260924 04:07:17.808344 21294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:17.808984 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:17.807404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48699 (local address 127.0.0.1:33351)
0924 04:07:17.807506 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:17.807510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:17.807688 (+ 178us) server_negotiation.cc:407] Connection header received
0924 04:07:17.807793 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:17.807796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:17.807829 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:17.807959 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:17.808633 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:17.808637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:17.808639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:17.808651 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:17.808676 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:17.808686 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:17.808752 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:17.808825 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 04:07:17.808863 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:07:17.809123 21294 log.cc:824] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:17.810983 21294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c: No bootstrap required, opened a new log
I20260924 04:07:17.814493 21294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } }
I20260924 04:07:17.814738 21294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:17.814797 21294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458cfb3e21ed49eca4cd8c4fecfbd67c, State: Initialized, Role: FOLLOWER
I20260924 04:07:17.815258 21294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } }
I20260924 04:07:17.815409 21294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:17.815475 21294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:17.815583 21294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:17.816526 21294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } }
I20260924 04:07:17.816845 21294 leader_election.cc:302] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 458cfb3e21ed49eca4cd8c4fecfbd67c; no voters:
I20260924 04:07:17.817128 21294 leader_election.cc:288] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:17.817224 21304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:17.817415 21304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [term 1 LEADER]: Becoming Leader. State: Replica: 458cfb3e21ed49eca4cd8c4fecfbd67c, State: Running, Role: LEADER
I20260924 04:07:17.817747 21304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } }
I20260924 04:07:17.818305 21294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:17.819499 21305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } } }
I20260924 04:07:17.819530 21306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458cfb3e21ed49eca4cd8c4fecfbd67c. Latest consensus state: current_term: 1 leader_uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458cfb3e21ed49eca4cd8c4fecfbd67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } } }
I20260924 04:07:17.819674 21306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:17.819659 21305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:17.820112 21314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:17.822916 21314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:17.828173 21314 catalog_manager.cc:1347] Generated new cluster ID: 5375ad6cc86649b8a1cc40d521a3a452
I20260924 04:07:17.828243 21314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:17.846047 21314 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:17.846448 21314 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:17.856233 21314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 458cfb3e21ed49eca4cd8c4fecfbd67c: Generated new TSK 0
I20260924 04:07:17.857132 21314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:17.864472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21226
2026-09-24T04:07:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:17 dist-test-slave-zcsk krb5kdc[21325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:17 dist-test-slave-zcsk krb5kdc[21325](info): set up 2 sockets
Sep 24 04:07:17 dist-test-slave-zcsk krb5kdc[21325](info): commencing operation
krb5kdc: starting...
W20260924 04:07:18.820034 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:18 dist-test-slave-zcsk krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:18.937063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:19.045568 21338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:19.045853 21338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:19.045913 21338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:19.050028 21338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:19.050105 21338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:19.050132 21338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:19.050153 21338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:19.050173 21338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:19.055626 21338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:19.056753 21338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:19.057839 21338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:19.066227 21343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:19.066227 21346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:19.066438 21338 server_base.cc:1034] running on GCE node
W20260924 04:07:19.066567 21344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:19.067167 21338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:19.068359 21338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:19.069537 21338 hybrid_clock.cc:648] HybridClock initialized: now 1790222839069505 us; error 46 us; skew 500 ppm
Sep 24 04:07:19 dist-test-slave-zcsk krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:19.072249 21338 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:19.073158 21338 webserver.cc:460] Webserver started at http://127.0.0.1:37305/ using document root <none> and password file <none>
I20260924 04:07:19.073688 21338 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:19.073745 21338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:19.073915 21338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:19.075964 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "39d46fdc11434e0483902e47c9bdf55c"
format_stamp: "Formatted at 2026-09-24 04:07:19 on dist-test-slave-zcsk"
server_key: "940ea0e2f5390a3894016254c8d83216"
server_key_iv: "5b730405987316de095c21b0048b8649"
server_key_version: "encryptionkey@0"
I20260924 04:07:19.076524 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "39d46fdc11434e0483902e47c9bdf55c"
format_stamp: "Formatted at 2026-09-24 04:07:19 on dist-test-slave-zcsk"
server_key: "940ea0e2f5390a3894016254c8d83216"
server_key_iv: "5b730405987316de095c21b0048b8649"
server_key_version: "encryptionkey@0"
I20260924 04:07:19.080200 21338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:07:19.082691 21353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:19.084009 21338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:19.084112 21338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "39d46fdc11434e0483902e47c9bdf55c"
format_stamp: "Formatted at 2026-09-24 04:07:19 on dist-test-slave-zcsk"
server_key: "940ea0e2f5390a3894016254c8d83216"
server_key_iv: "5b730405987316de095c21b0048b8649"
server_key_version: "encryptionkey@0"
I20260924 04:07:19.084225 21338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:19.108713 21338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:19.111874 21338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:19.112077 21338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:19.118788 21338 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46501
I20260924 04:07:19.118820 21405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46501 every 8 connection(s)
I20260924 04:07:19.119733 21338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:19.122453 21406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:19.123251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21338
I20260924 04:07:19.123379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:19.123677 392 external_mini_cluster.cc:1428] Setting key be248ac8df132012be2b487ee2f2183c
I20260924 04:07:19.129129 21406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c: Bootstrap starting.
I20260924 04:07:19.131877 21406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c: Neither blocks nor log segments found. Creating new log.
Sep 24 04:07:19 dist-test-slave-zcsk krb5kdc[21325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:07:19.132638 21406 log.cc:824] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:19.134752 21406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c: No bootstrap required, opened a new log
I20260924 04:07:19.137863 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:19.125002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:34553 (local address 127.0.0.1:46501)
0924 04:07:19.125413 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:19.125429 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:19.125465 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:07:19.126111 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:19.126128 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:19.126480 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:19.126760 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:19.127651 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:19.129120 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:19.130117 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:19.130604 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:19.130756 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:19.133167 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:19.133172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:19.133190 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:19.133220 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:19.134940 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:19.135475 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:19.135481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:19.135486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:19.135609 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:19.135957 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:19.135960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:19.135962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:19.136393 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:19.136588 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:19.136905 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 04:07:19.137123 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":124,"threads_started":1}
I20260924 04:07:19.138523 21406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d46fdc11434e0483902e47c9bdf55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46501 } }
I20260924 04:07:19.138775 21406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:19.138827 21406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d46fdc11434e0483902e47c9bdf55c, State: Initialized, Role: FOLLOWER
I20260924 04:07:19.139293 21406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d46fdc11434e0483902e47c9bdf55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46501 } }
I20260924 04:07:19.139436 21406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:19.139488 21406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:19.139568 21406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:19.140694 21406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d46fdc11434e0483902e47c9bdf55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46501 } }
I20260924 04:07:19.141041 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:19.141075 21406 leader_election.cc:302] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39d46fdc11434e0483902e47c9bdf55c; no voters:
I20260924 04:07:19.141397 21406 leader_election.cc:288] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:19.141487 21411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:19.141656 21411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [term 1 LEADER]: Becoming Leader. State: Replica: 39d46fdc11434e0483902e47c9bdf55c, State: Running, Role: LEADER
I20260924 04:07:19.142014 21411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d46fdc11434e0483902e47c9bdf55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46501 } }
I20260924 04:07:19.142706 21406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:19.143270 21413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39d46fdc11434e0483902e47c9bdf55c. Latest consensus state: current_term: 1 leader_uuid: "39d46fdc11434e0483902e47c9bdf55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d46fdc11434e0483902e47c9bdf55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46501 } } }
I20260924 04:07:19.143412 21413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:19.143666 21412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39d46fdc11434e0483902e47c9bdf55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d46fdc11434e0483902e47c9bdf55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46501 } } }
I20260924 04:07:19.143754 21412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:19.143925 21424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:19.147414 21424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:19.154031 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:19.143052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:54457 (local address 127.0.0.1:46501)
0924 04:07:19.143144 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:19.143148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:19.144326 (+ 1178us) server_negotiation.cc:407] Connection header received
0924 04:07:19.144529 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:19.144532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:19.144581 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:19.144627 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:19.146195 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:19.147002 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:19.148281 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:19.148530 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:19.148618 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:19.150121 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:19.150125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:19.150128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:19.150156 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:19.151861 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:19.152769 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:19.152772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:19.152774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:19.152858 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:19.153179 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:19.153182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:19.153184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:19.153367 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:19.153502 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:19.153768 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 04:07:19.153862 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 04:07:19.154420 21424 catalog_manager.cc:1347] Generated new cluster ID: 24461896c2274a4087077aededdbafe4
I20260924 04:07:19.154479 21424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:19.161461 21424 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:19.162340 21424 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:19.167884 21424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39d46fdc11434e0483902e47c9bdf55c: Generated new TSK 0
I20260924 04:07:19.168576 21424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:19.176534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21338
W20260924 04:07:19.181231 21323 connection.cc:535] client connection to 127.0.0.1:46501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46501): Connection reset by peer (error 104)
2026-09-24T04:07:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:19 dist-test-slave-zcsk krb5kdc[21437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:19 dist-test-slave-zcsk krb5kdc[21437](info): set up 2 sockets
Sep 24 04:07:19 dist-test-slave-zcsk krb5kdc[21437](info): commencing operation
krb5kdc: starting...
W20260924 04:07:20.148343 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:20 dist-test-slave-zcsk krb5kdc[21437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:20.267773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:20.377746 21450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:20.378074 21450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:20.378137 21450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:20.382310 21450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:20.382400 21450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:20.382428 21450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:20.382448 21450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:20.382470 21450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:20.388098 21450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:20.389248 21450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:20.390357 21450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:20.398934 21450 server_base.cc:1034] running on GCE node
W20260924 04:07:20.398953 21455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:20.398954 21458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:20.398923 21456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:20.399725 21450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:20.400885 21450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:20.402047 21450 hybrid_clock.cc:648] HybridClock initialized: now 1790222840402027 us; error 32 us; skew 500 ppm
Sep 24 04:07:20 dist-test-slave-zcsk krb5kdc[21437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:20.408660 21450 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:20.409693 21450 webserver.cc:460] Webserver started at http://127.0.0.1:45907/ using document root <none> and password file <none>
I20260924 04:07:20.410318 21450 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:20.410392 21450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:20.410594 21450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:20.412737 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "171f94ca6e9244168c29b23fb803a1e1"
format_stamp: "Formatted at 2026-09-24 04:07:20 on dist-test-slave-zcsk"
server_key: "20f35e3a41ad90de124dcfaa84f48d3b"
server_key_iv: "6d468194412f716fa9ca34a2d2a9794c"
server_key_version: "encryptionkey@0"
I20260924 04:07:20.413286 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "171f94ca6e9244168c29b23fb803a1e1"
format_stamp: "Formatted at 2026-09-24 04:07:20 on dist-test-slave-zcsk"
server_key: "20f35e3a41ad90de124dcfaa84f48d3b"
server_key_iv: "6d468194412f716fa9ca34a2d2a9794c"
server_key_version: "encryptionkey@0"
I20260924 04:07:20.417273 21450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:07:20.419832 21465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:20.421378 21450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:20.421506 21450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "171f94ca6e9244168c29b23fb803a1e1"
format_stamp: "Formatted at 2026-09-24 04:07:20 on dist-test-slave-zcsk"
server_key: "20f35e3a41ad90de124dcfaa84f48d3b"
server_key_iv: "6d468194412f716fa9ca34a2d2a9794c"
server_key_version: "encryptionkey@0"
I20260924 04:07:20.421619 21450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:20.433260 21450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:20.436249 21450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:20.436434 21450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:20.443490 21450 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38971
I20260924 04:07:20.443511 21517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38971 every 8 connection(s)
I20260924 04:07:20.444545 21450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:20.447515 21518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:20.453399 21518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1: Bootstrap starting.
I20260924 04:07:20.454361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21450
I20260924 04:07:20.454483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:20.454742 392 external_mini_cluster.cc:1428] Setting key 0ad974106b87baf43867e580aedea711
I20260924 04:07:20.456313 21518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:20.457056 21518 log.cc:824] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:20.458951 21518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1: No bootstrap required, opened a new log
I20260924 04:07:20.462724 21518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171f94ca6e9244168c29b23fb803a1e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } }
I20260924 04:07:20.463054 21518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:20.463120 21518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171f94ca6e9244168c29b23fb803a1e1, State: Initialized, Role: FOLLOWER
I20260924 04:07:20.463631 21518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171f94ca6e9244168c29b23fb803a1e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } }
I20260924 04:07:20.463762 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:20.455861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:58949 (local address 127.0.0.1:38971)
0924 04:07:20.456329 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:20.456346 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:20.456383 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:07:20.457090 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:20.457109 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:20.457506 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:20.457829 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:20.458264 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:20.459285 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:20.460198 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:20.460703 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:20.460860 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:20.461850 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:20.461855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:20.461870 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:20.462344 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:20.462402 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:20.462426 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:20.462542 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:20.462740 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 04:07:20.462882 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
I20260924 04:07:20.463766 21518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:20.463901 21518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:20.464039 21518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:20.465229 21518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171f94ca6e9244168c29b23fb803a1e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } }
I20260924 04:07:20.465572 21518 leader_election.cc:302] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 171f94ca6e9244168c29b23fb803a1e1; no voters:
I20260924 04:07:20.465916 21518 leader_election.cc:288] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:20.466045 21523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:20.466306 21523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [term 1 LEADER]: Becoming Leader. State: Replica: 171f94ca6e9244168c29b23fb803a1e1, State: Running, Role: LEADER
I20260924 04:07:20.466727 21523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171f94ca6e9244168c29b23fb803a1e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } }
I20260924 04:07:20.467181 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:20.467334 21518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:20.468523 21525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 171f94ca6e9244168c29b23fb803a1e1. Latest consensus state: current_term: 1 leader_uuid: "171f94ca6e9244168c29b23fb803a1e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171f94ca6e9244168c29b23fb803a1e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } } }
I20260924 04:07:20.468521 21524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "171f94ca6e9244168c29b23fb803a1e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171f94ca6e9244168c29b23fb803a1e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38971 } } }
I20260924 04:07:20.468681 21525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:20.468681 21524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:20.469143 21536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:20.472549 21536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:20.474476 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:20.469457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:47035 (local address 127.0.0.1:38971)
0924 04:07:20.469719 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:20.469724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:20.470282 (+ 558us) server_negotiation.cc:407] Connection header received
0924 04:07:20.470383 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:20.470387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:20.470434 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:20.470495 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:20.470753 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:20.471484 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:20.472648 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:20.472975 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:20.473083 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:20.474000 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:20.474003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:20.474006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:20.474019 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:20.474050 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:20.474065 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:20.474156 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:20.474295 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 04:07:20.474339 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260924 04:07:20.478056 21536 catalog_manager.cc:1347] Generated new cluster ID: 5685d26d530c46a7a63ee5fcec8586e3
I20260924 04:07:20.478128 21536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:20.508665 21536 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:20.509660 21536 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:20.516335 21536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 171f94ca6e9244168c29b23fb803a1e1: Generated new TSK 0
I20260924 04:07:20.517205 21536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:20.530507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21450
W20260924 04:07:20.535234 21435 connection.cc:535] client connection to 127.0.0.1:38971 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38971): Connection reset by peer (error 104)
2026-09-24T04:07:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:20 dist-test-slave-zcsk krb5kdc[21549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:20 dist-test-slave-zcsk krb5kdc[21549](info): set up 2 sockets
Sep 24 04:07:20 dist-test-slave-zcsk krb5kdc[21549](info): commencing operation
krb5kdc: starting...
W20260924 04:07:21.496042 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:21 dist-test-slave-zcsk krb5kdc[21549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:21.615681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:21.724696 21562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:21.724964 21562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:21.725005 21562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:21.728947 21562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:21.729007 21562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:21.729025 21562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:21.729040 21562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:21.729054 21562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:21.734264 21562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:21.735257 21562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:21.736227 21562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:21.744431 21562 server_base.cc:1034] running on GCE node
W20260924 04:07:21.744395 21570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:21.744374 21568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:21.744377 21567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:21.745124 21562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:21.746276 21562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:21.747462 21562 hybrid_clock.cc:648] HybridClock initialized: now 1790222841747447 us; error 27 us; skew 500 ppm
Sep 24 04:07:21 dist-test-slave-zcsk krb5kdc[21549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:21.750329 21562 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:21.751322 21562 webserver.cc:460] Webserver started at http://127.0.0.1:43289/ using document root <none> and password file <none>
I20260924 04:07:21.751866 21562 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:21.751955 21562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:21.752166 21562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:21.754196 21562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "c4d11a779cc846c780078a647ac417ab"
format_stamp: "Formatted at 2026-09-24 04:07:21 on dist-test-slave-zcsk"
server_key: "c9bf07efb33ae61756d7e7609e18535b"
server_key_iv: "9de173909eee09cc68902b38d37c02bc"
server_key_version: "encryptionkey@0"
I20260924 04:07:21.754747 21562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4d11a779cc846c780078a647ac417ab"
format_stamp: "Formatted at 2026-09-24 04:07:21 on dist-test-slave-zcsk"
server_key: "c9bf07efb33ae61756d7e7609e18535b"
server_key_iv: "9de173909eee09cc68902b38d37c02bc"
server_key_version: "encryptionkey@0"
I20260924 04:07:21.758571 21562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 04:07:21.761081 21577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:21.762445 21562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:07:21.762562 21562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "c4d11a779cc846c780078a647ac417ab"
format_stamp: "Formatted at 2026-09-24 04:07:21 on dist-test-slave-zcsk"
server_key: "c9bf07efb33ae61756d7e7609e18535b"
server_key_iv: "9de173909eee09cc68902b38d37c02bc"
server_key_version: "encryptionkey@0"
I20260924 04:07:21.762679 21562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:21.767294 21562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:21.770344 21562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:21.770509 21562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:21.777331 21562 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34371
I20260924 04:07:21.777386 21629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34371 every 8 connection(s)
I20260924 04:07:21.778304 21562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:21.781162 21630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:21.781630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21562
I20260924 04:07:21.781759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:21.782018 392 external_mini_cluster.cc:1428] Setting key e3952dc59910cc3d7cfdcd4ab4327971
I20260924 04:07:21.787776 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:21.783416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:42061 (local address 127.0.0.1:34371)
0924 04:07:21.783862 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:21.783883 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:21.783965 (+ 82us) server_negotiation.cc:407] Connection header received
0924 04:07:21.784819 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:21.784843 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:21.785244 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:21.785624 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:21.786349 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:21.786355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:21.786371 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:21.786382 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:21.786442 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:21.786468 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:21.786609 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:21.786677 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 04:07:21.786844 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260924 04:07:21.788095 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab: Bootstrap starting.
I20260924 04:07:21.790330 21630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:21.790527 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:21.791056 21630 log.cc:824] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:21.793171 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab: No bootstrap required, opened a new log
I20260924 04:07:21.793999 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:21.792470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:41931 (local address 127.0.0.1:34371)
0924 04:07:21.792588 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:21.792593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:21.792760 (+ 167us) server_negotiation.cc:407] Connection header received
0924 04:07:21.792886 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:21.792888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:21.792915 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:21.792974 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:21.793654 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:21.793656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:21.793658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:21.793668 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:21.793691 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:21.793708 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:21.793771 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:21.793863 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 04:07:21.793894 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:07:21.797152 21630 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d11a779cc846c780078a647ac417ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260924 04:07:21.797497 21630 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:21.797573 21630 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d11a779cc846c780078a647ac417ab, State: Initialized, Role: FOLLOWER
I20260924 04:07:21.798074 21630 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d11a779cc846c780078a647ac417ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260924 04:07:21.798240 21630 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:21.798301 21630 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:21.798400 21630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:21.799612 21630 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d11a779cc846c780078a647ac417ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260924 04:07:21.800033 21630 leader_election.cc:302] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d11a779cc846c780078a647ac417ab; no voters:
I20260924 04:07:21.800401 21630 leader_election.cc:288] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:21.800506 21640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:21.800716 21640 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [term 1 LEADER]: Becoming Leader. State: Replica: c4d11a779cc846c780078a647ac417ab, State: Running, Role: LEADER
I20260924 04:07:21.801077 21640 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d11a779cc846c780078a647ac417ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260924 04:07:21.801841 21630 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:21.802815 21641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d11a779cc846c780078a647ac417ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d11a779cc846c780078a647ac417ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } } }
I20260924 04:07:21.802948 21641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:21.804821 21642 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d11a779cc846c780078a647ac417ab. Latest consensus state: current_term: 1 leader_uuid: "c4d11a779cc846c780078a647ac417ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d11a779cc846c780078a647ac417ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } } }
I20260924 04:07:21.804920 21642 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab [sys.catalog]: This master's current role is: LEADER
W20260924 04:07:21.807401 21655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:21.807480 21655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:21.807547 21649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:21.808461 21649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:21.816090 21649 catalog_manager.cc:1347] Generated new cluster ID: f6879110386d4594ab39bfe4c286827a
I20260924 04:07:21.816174 21649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:21.834005 21649 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:21.834419 21649 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:21.843420 21649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d11a779cc846c780078a647ac417ab: Generated new TSK 0
I20260924 04:07:21.844197 21649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:21.847543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21562
2026-09-24T04:07:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:21 dist-test-slave-zcsk krb5kdc[21661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:21 dist-test-slave-zcsk krb5kdc[21661](info): set up 2 sockets
Sep 24 04:07:21 dist-test-slave-zcsk krb5kdc[21661](info): commencing operation
krb5kdc: starting...
W20260924 04:07:22.804226 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:22 dist-test-slave-zcsk krb5kdc[21661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:22.927151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:23.039244 21674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:23.039538 21674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:23.039599 21674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:23.043650 21674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:23.043726 21674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:23.043754 21674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:23.043776 21674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:23.043797 21674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:23.049316 21674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:23.050462 21674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:23.051540 21674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:23.061231 21674 server_base.cc:1034] running on GCE node
W20260924 04:07:23.061308 21680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:23.061328 21679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:23.061206 21682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:23.062124 21674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:23.063356 21674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:23.064527 21674 hybrid_clock.cc:648] HybridClock initialized: now 1790222843064489 us; error 50 us; skew 500 ppm
Sep 24 04:07:23 dist-test-slave-zcsk krb5kdc[21661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:23.067685 21674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:23.068795 21674 webserver.cc:460] Webserver started at http://127.0.0.1:40229/ using document root <none> and password file <none>
I20260924 04:07:23.069417 21674 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:23.069499 21674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:23.069727 21674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:23.071952 21674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "5d2fc54426634feeb721db2c4127facd"
format_stamp: "Formatted at 2026-09-24 04:07:23 on dist-test-slave-zcsk"
server_key: "b5de854c8b9d7c37a5012d6a00988ce7"
server_key_iv: "59fbb877e339c06a973037503e25fe3e"
server_key_version: "encryptionkey@0"
I20260924 04:07:23.072515 21674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d2fc54426634feeb721db2c4127facd"
format_stamp: "Formatted at 2026-09-24 04:07:23 on dist-test-slave-zcsk"
server_key: "b5de854c8b9d7c37a5012d6a00988ce7"
server_key_iv: "59fbb877e339c06a973037503e25fe3e"
server_key_version: "encryptionkey@0"
I20260924 04:07:23.076568 21674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:07:23.079247 21689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:23.080794 21674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:23.080930 21674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "5d2fc54426634feeb721db2c4127facd"
format_stamp: "Formatted at 2026-09-24 04:07:23 on dist-test-slave-zcsk"
server_key: "b5de854c8b9d7c37a5012d6a00988ce7"
server_key_iv: "59fbb877e339c06a973037503e25fe3e"
server_key_version: "encryptionkey@0"
I20260924 04:07:23.081048 21674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:23.109303 21674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:23.112473 21674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:23.112658 21674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:23.119518 21674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33867
I20260924 04:07:23.119571 21741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33867 every 8 connection(s)
I20260924 04:07:23.120576 21674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:23.123517 21742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:23.123962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21674
I20260924 04:07:23.124104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:23.124447 392 external_mini_cluster.cc:1428] Setting key 9ff4af66a1b7561d8f2b07402ab2a6cd
I20260924 04:07:23.129628 21742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd: Bootstrap starting.
I20260924 04:07:23.131899 21742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:23.132658 21742 log.cc:824] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:07:23 dist-test-slave-zcsk krb5kdc[21661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:07:23.134650 21742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd: No bootstrap required, opened a new log
I20260924 04:07:23.138263 21742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2fc54426634feeb721db2c4127facd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260924 04:07:23.138510 21742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:23.138569 21742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2fc54426634feeb721db2c4127facd, State: Initialized, Role: FOLLOWER
I20260924 04:07:23.138725 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:23.125935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:38453 (local address 127.0.0.1:33867)
0924 04:07:23.126415 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:23.126430 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:23.126471 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:07:23.127128 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:23.127145 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:23.127558 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:23.127964 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:23.128729 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:23.129783 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:23.130644 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:23.131119 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:23.131261 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:23.133944 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:23.133949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:23.133963 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:23.133989 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:23.135641 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:23.136184 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:23.136189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:23.136194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:23.136347 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:23.136669 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:23.136672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:23.136673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:23.137097 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:23.137334 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:23.137647 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 04:07:23.137866 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":136,"threads_started":1}
I20260924 04:07:23.139041 21742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2fc54426634feeb721db2c4127facd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260924 04:07:23.139206 21742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:23.139269 21742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:23.139362 21742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:23.140473 21742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2fc54426634feeb721db2c4127facd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260924 04:07:23.140836 21742 leader_election.cc:302] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d2fc54426634feeb721db2c4127facd; no voters:
I20260924 04:07:23.141160 21742 leader_election.cc:288] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:23.141249 21747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:23.141434 21747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [term 1 LEADER]: Becoming Leader. State: Replica: 5d2fc54426634feeb721db2c4127facd, State: Running, Role: LEADER
I20260924 04:07:23.141747 21747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2fc54426634feeb721db2c4127facd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260924 04:07:23.142136 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:23.142500 21742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:23.144245 21748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d2fc54426634feeb721db2c4127facd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2fc54426634feeb721db2c4127facd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } } }
I20260924 04:07:23.144377 21748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:23.144763 21749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d2fc54426634feeb721db2c4127facd. Latest consensus state: current_term: 1 leader_uuid: "5d2fc54426634feeb721db2c4127facd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2fc54426634feeb721db2c4127facd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } } }
I20260924 04:07:23.144881 21749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:23.144747 21765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:23.148577 21765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:23.154250 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:23.145348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:39131 (local address 127.0.0.1:33867)
0924 04:07:23.145509 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:23.145514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:23.146125 (+ 611us) server_negotiation.cc:407] Connection header received
0924 04:07:23.146244 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:23.146247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:23.146304 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:23.146393 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:23.147208 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:23.147733 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:23.148602 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:23.148878 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:23.148968 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:23.150299 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:23.150303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:23.150305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:23.150340 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:23.152079 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:23.152650 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:23.152653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:23.152656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:23.152751 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:23.153176 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:23.153179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:23.153181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:23.153384 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:23.153527 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:23.153994 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 04:07:23.154097 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 04:07:23.155064 21765 catalog_manager.cc:1347] Generated new cluster ID: 96eea40acebd44fcb7757008114a499c
I20260924 04:07:23.155189 21765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:23.164373 21765 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:23.165657 21765 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:23.173187 21765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d2fc54426634feeb721db2c4127facd: Generated new TSK 0
I20260924 04:07:23.173923 21765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:23.182701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21674
W20260924 04:07:23.187937 21659 connection.cc:535] client connection to 127.0.0.1:33867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33867): Connection reset by peer (error 104)
2026-09-24T04:07:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:23 dist-test-slave-zcsk krb5kdc[21773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:23 dist-test-slave-zcsk krb5kdc[21773](info): set up 2 sockets
Sep 24 04:07:23 dist-test-slave-zcsk krb5kdc[21773](info): commencing operation
krb5kdc: starting...
W20260924 04:07:24.148444 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:24 dist-test-slave-zcsk krb5kdc[21773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:24.270782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:24.382372 21786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:24.382660 21786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:24.382709 21786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:24.387128 21786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:24.387204 21786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:24.387228 21786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:24.387246 21786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:24.387264 21786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:24.392877 21786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:24.393930 21786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:24.394955 21786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:24.403795 21786 server_base.cc:1034] running on GCE node
W20260924 04:07:24.403705 21791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:24.403713 21794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:24.403705 21792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:24.404511 21786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:24.405643 21786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:24.406813 21786 hybrid_clock.cc:648] HybridClock initialized: now 1790222844406789 us; error 36 us; skew 500 ppm
Sep 24 04:07:24 dist-test-slave-zcsk krb5kdc[21773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:24.409588 21786 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:24.410620 21786 webserver.cc:460] Webserver started at http://127.0.0.1:36759/ using document root <none> and password file <none>
I20260924 04:07:24.411206 21786 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:24.411293 21786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:24.411509 21786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:24.413841 21786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "0fe04876e02149928f1c26b648ffa1ef"
format_stamp: "Formatted at 2026-09-24 04:07:24 on dist-test-slave-zcsk"
server_key: "2e6cbbb61d09063134d533695cf23e6f"
server_key_iv: "d2ac5461e4e96fb42129f14362623545"
server_key_version: "encryptionkey@0"
I20260924 04:07:24.414451 21786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "0fe04876e02149928f1c26b648ffa1ef"
format_stamp: "Formatted at 2026-09-24 04:07:24 on dist-test-slave-zcsk"
server_key: "2e6cbbb61d09063134d533695cf23e6f"
server_key_iv: "d2ac5461e4e96fb42129f14362623545"
server_key_version: "encryptionkey@0"
I20260924 04:07:24.418522 21786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 04:07:24.421185 21801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:24.422574 21786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:07:24.422746 21786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "0fe04876e02149928f1c26b648ffa1ef"
format_stamp: "Formatted at 2026-09-24 04:07:24 on dist-test-slave-zcsk"
server_key: "2e6cbbb61d09063134d533695cf23e6f"
server_key_iv: "d2ac5461e4e96fb42129f14362623545"
server_key_version: "encryptionkey@0"
I20260924 04:07:24.422870 21786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:24.449596 21786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:24.453011 21786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:24.453218 21786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:24.460750 21786 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34411
I20260924 04:07:24.460801 21853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34411 every 8 connection(s)
I20260924 04:07:24.461822 21786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:24.464887 21854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:24.467329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21786
I20260924 04:07:24.467463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:24.467713 392 external_mini_cluster.cc:1428] Setting key 0446919c37232c1b1eff194376d81445
I20260924 04:07:24.471369 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef: Bootstrap starting.
I20260924 04:07:24.474013 21854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:24.474865 21854 log.cc:824] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:24.477157 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef: No bootstrap required, opened a new log
I20260924 04:07:24.478011 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:24.469250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:51007 (local address 127.0.0.1:34411)
0924 04:07:24.469697 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:24.469716 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:24.469759 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:07:24.470593 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:24.470618 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:24.471081 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:24.471453 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:24.471877 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:24.473359 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:24.474317 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:24.475014 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:24.475169 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:24.476031 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:24.476038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:24.476055 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:24.476535 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:24.476607 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:24.476636 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:24.476760 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:24.476886 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:07:24.477028 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260924 04:07:24.480782 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:24.481014 21854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe04876e02149928f1c26b648ffa1ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260924 04:07:24.481243 21854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:24.481293 21854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe04876e02149928f1c26b648ffa1ef, State: Initialized, Role: FOLLOWER
I20260924 04:07:24.481688 21854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe04876e02149928f1c26b648ffa1ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260924 04:07:24.481803 21854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:24.481859 21854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:24.481956 21854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:24.482867 21854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe04876e02149928f1c26b648ffa1ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260924 04:07:24.483183 21854 leader_election.cc:302] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fe04876e02149928f1c26b648ffa1ef; no voters:
I20260924 04:07:24.483477 21854 leader_election.cc:288] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:24.483587 21864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:24.483803 21864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [term 1 LEADER]: Becoming Leader. State: Replica: 0fe04876e02149928f1c26b648ffa1ef, State: Running, Role: LEADER
I20260924 04:07:24.484216 21864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe04876e02149928f1c26b648ffa1ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260924 04:07:24.484723 21854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:24.485296 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fe04876e02149928f1c26b648ffa1ef. Latest consensus state: current_term: 1 leader_uuid: "0fe04876e02149928f1c26b648ffa1ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe04876e02149928f1c26b648ffa1ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } } }
I20260924 04:07:24.485410 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:24.485749 21865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fe04876e02149928f1c26b648ffa1ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe04876e02149928f1c26b648ffa1ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } } }
I20260924 04:07:24.485846 21865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:24.486164 21869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:24.487749 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:24.482762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:51215 (local address 127.0.0.1:34411)
0924 04:07:24.482881 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:24.482885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:24.483065 (+ 180us) server_negotiation.cc:407] Connection header received
0924 04:07:24.483189 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:24.483191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:24.483236 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:24.483299 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:24.483588 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:24.484363 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:24.485466 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:24.485793 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:24.485869 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:24.487001 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:24.487005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:24.487008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:24.487022 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:24.487049 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:24.487062 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:24.487147 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:24.487584 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 04:07:24.487619 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:07:24.489413 21869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:24.494657 21869 catalog_manager.cc:1347] Generated new cluster ID: bae6f65d6d7b447f8e4c9219219e127e
I20260924 04:07:24.494730 21869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:24.511801 21869 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:24.513164 21869 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:24.526937 21869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fe04876e02149928f1c26b648ffa1ef: Generated new TSK 0
I20260924 04:07:24.527850 21869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:24.535799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21786
W20260924 04:07:24.540683 21771 connection.cc:535] client connection to 127.0.0.1:34411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34411): Connection reset by peer (error 104)
2026-09-24T04:07:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:24 dist-test-slave-zcsk krb5kdc[21885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:24 dist-test-slave-zcsk krb5kdc[21885](info): set up 2 sockets
Sep 24 04:07:24 dist-test-slave-zcsk krb5kdc[21885](info): commencing operation
krb5kdc: starting...
W20260924 04:07:25.491807 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:25 dist-test-slave-zcsk krb5kdc[21885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:07:25.611577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:25.721915 21898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:25.722196 21898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:25.722239 21898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:25.726287 21898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:25.726362 21898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:25.726380 21898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:25.726397 21898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:25.726410 21898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:25.731812 21898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:25.732897 21898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:25.733887 21898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:25.742323 21898 server_base.cc:1034] running on GCE node
W20260924 04:07:25.742269 21903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:25.742287 21906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:25.742269 21904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:25.743144 21898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:25.744307 21898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:25.745481 21898 hybrid_clock.cc:648] HybridClock initialized: now 1790222845745462 us; error 31 us; skew 500 ppm
Sep 24 04:07:25 dist-test-slave-zcsk krb5kdc[21885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:25.751937 21898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:07:25.752955 21898 webserver.cc:460] Webserver started at http://127.0.0.1:37109/ using document root <none> and password file <none>
I20260924 04:07:25.753530 21898 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:25.753608 21898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:25.753823 21898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:25.755980 21898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "5b8403ef35104877acb381645e923da0"
format_stamp: "Formatted at 2026-09-24 04:07:25 on dist-test-slave-zcsk"
server_key: "0ef2e7ad70f5cdb28841f2ad77331cc8"
server_key_iv: "248224739831eed9adcbc1aeb0f71935"
server_key_version: "encryptionkey@0"
I20260924 04:07:25.756582 21898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b8403ef35104877acb381645e923da0"
format_stamp: "Formatted at 2026-09-24 04:07:25 on dist-test-slave-zcsk"
server_key: "0ef2e7ad70f5cdb28841f2ad77331cc8"
server_key_iv: "248224739831eed9adcbc1aeb0f71935"
server_key_version: "encryptionkey@0"
I20260924 04:07:25.760546 21898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:07:25.763128 21913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:25.764493 21898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:25.764652 21898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "5b8403ef35104877acb381645e923da0"
format_stamp: "Formatted at 2026-09-24 04:07:25 on dist-test-slave-zcsk"
server_key: "0ef2e7ad70f5cdb28841f2ad77331cc8"
server_key_iv: "248224739831eed9adcbc1aeb0f71935"
server_key_version: "encryptionkey@0"
I20260924 04:07:25.764765 21898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:25.769657 21898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:25.772876 21898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:25.773065 21898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:25.779942 21898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38453
I20260924 04:07:25.780000 21965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38453 every 8 connection(s)
I20260924 04:07:25.780892 21898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:25.783684 21966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:25.787801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 21898
I20260924 04:07:25.787966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:25.788254 392 external_mini_cluster.cc:1428] Setting key 24d8cd875adfe798a26bd8875d1936e2
I20260924 04:07:25.789769 21966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0: Bootstrap starting.
I20260924 04:07:25.792549 21966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:25.793447 21966 log.cc:824] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:25.793943 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:25.789599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:60099 (local address 127.0.0.1:38453)
0924 04:07:25.790052 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:25.790071 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:25.790124 (+ 53us) server_negotiation.cc:407] Connection header received
0924 04:07:25.790954 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:25.790980 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:25.791372 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:25.791732 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:25.792466 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:25.792472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:25.792488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:25.792500 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:25.792569 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:25.792599 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:25.792709 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:25.792875 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 04:07:25.793040 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":146,"threads_started":1}
I20260924 04:07:25.795404 21966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0: No bootstrap required, opened a new log
I20260924 04:07:25.796671 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:25.799297 21966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8403ef35104877acb381645e923da0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } }
I20260924 04:07:25.799530 21966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:25.799607 21966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8403ef35104877acb381645e923da0, State: Initialized, Role: FOLLOWER
I20260924 04:07:25.800184 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:25.798590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.36:51399 (local address 127.0.0.1:38453)
0924 04:07:25.798704 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:25.798708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:25.798878 (+ 170us) server_negotiation.cc:407] Connection header received
0924 04:07:25.798965 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:25.798968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:25.799002 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:25.799081 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:25.799771 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:25.799774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:25.799777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:25.799790 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:25.799818 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:25.799831 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:25.799936 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:25.800013 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 04:07:25.800051 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:07:25.800127 21966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8403ef35104877acb381645e923da0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } }
I20260924 04:07:25.800279 21966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:25.800348 21966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:25.800446 21966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:25.801450 21966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8403ef35104877acb381645e923da0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } }
I20260924 04:07:25.801805 21966 leader_election.cc:302] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b8403ef35104877acb381645e923da0; no voters:
I20260924 04:07:25.802088 21966 leader_election.cc:288] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:25.802224 21976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:25.802443 21976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [term 1 LEADER]: Becoming Leader. State: Replica: 5b8403ef35104877acb381645e923da0, State: Running, Role: LEADER
I20260924 04:07:25.802774 21976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8403ef35104877acb381645e923da0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } }
I20260924 04:07:25.803310 21966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:25.804412 21978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b8403ef35104877acb381645e923da0. Latest consensus state: current_term: 1 leader_uuid: "5b8403ef35104877acb381645e923da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8403ef35104877acb381645e923da0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } } }
I20260924 04:07:25.804562 21978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:25.804751 21977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b8403ef35104877acb381645e923da0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8403ef35104877acb381645e923da0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38453 } } }
I20260924 04:07:25.805069 21977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0 [sys.catalog]: This master's current role is: LEADER
W20260924 04:07:25.807783 21991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:25.807857 21991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:25.807951 21987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:25.808677 21987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:25.814529 21987 catalog_manager.cc:1347] Generated new cluster ID: a6e6115a06804f1491a83828f0f5ca3f
I20260924 04:07:25.814631 21987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:25.839062 21987 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:25.839464 21987 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:25.848613 21987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b8403ef35104877acb381645e923da0: Generated new TSK 0
I20260924 04:07:25.849505 21987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:25.853677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 21898
2026-09-24T04:07:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:25 dist-test-slave-zcsk krb5kdc[21997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:25 dist-test-slave-zcsk krb5kdc[21997](info): set up 2 sockets
Sep 24 04:07:25 dist-test-slave-zcsk krb5kdc[21997](info): commencing operation
krb5kdc: starting...
W20260924 04:07:26.808041 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:26 dist-test-slave-zcsk krb5kdc[21997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:26.928472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39617
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:27.040697 22010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:27.040987 22010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:27.041030 22010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:27.045086 22010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:27.045161 22010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:27.045179 22010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:27.045204 22010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:27.045233 22010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:27.050905 22010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:27.052143 22010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:27.053179 22010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:27.061924 22018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:27.062079 22015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:27.061975 22016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:27.061968 22010 server_base.cc:1034] running on GCE node
I20260924 04:07:27.062817 22010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:27.064037 22010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:27.065227 22010 hybrid_clock.cc:648] HybridClock initialized: now 1790222847065204 us; error 36 us; skew 500 ppm
Sep 24 04:07:27 dist-test-slave-zcsk krb5kdc[21997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:27.068140 22010 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:27.069108 22010 webserver.cc:460] Webserver started at http://127.0.98.62:45099/ using document root <none> and password file <none>
I20260924 04:07:27.069666 22010 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:27.069732 22010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:27.069933 22010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:27.072084 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c"
format_stamp: "Formatted at 2026-09-24 04:07:27 on dist-test-slave-zcsk"
server_key: "b09b4015971ad4348a9098030f2e868d"
server_key_iv: "563e3e40a902dcafa06cbeb9a8585cf2"
server_key_version: "encryptionkey@0"
I20260924 04:07:27.072636 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c"
format_stamp: "Formatted at 2026-09-24 04:07:27 on dist-test-slave-zcsk"
server_key: "b09b4015971ad4348a9098030f2e868d"
server_key_iv: "563e3e40a902dcafa06cbeb9a8585cf2"
server_key_version: "encryptionkey@0"
I20260924 04:07:27.076467 22010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:07:27.079084 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:27.080482 22010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:27.080602 22010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c"
format_stamp: "Formatted at 2026-09-24 04:07:27 on dist-test-slave-zcsk"
server_key: "b09b4015971ad4348a9098030f2e868d"
server_key_iv: "563e3e40a902dcafa06cbeb9a8585cf2"
server_key_version: "encryptionkey@0"
I20260924 04:07:27.080709 22010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:27.108490 22010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:27.111548 22010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:27.111743 22010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:27.118494 22010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39617
I20260924 04:07:27.118547 22077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39617 every 8 connection(s)
I20260924 04:07:27.119447 22010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:27.122270 22078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:27.125063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22010
I20260924 04:07:27.125209 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:27.125499 392 external_mini_cluster.cc:1428] Setting key 9ab16a3fbd30fe1ea0bab2292504aca7
I20260924 04:07:27.128345 22078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c: Bootstrap starting.
I20260924 04:07:27.130822 22078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:27.131719 22078 log.cc:824] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:27.133586 22078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c: No bootstrap required, opened a new log
Sep 24 04:07:27 dist-test-slave-zcsk krb5kdc[21997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:27.136868 22078 raft_consensus.cc:348] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } }
I20260924 04:07:27.137104 22078 raft_consensus.cc:374] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:27.137162 22078 raft_consensus.cc:729] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98c0ae22a7f4ce3bb3c9da3753ab51c, State: Initialized, Role: FOLLOWER
I20260924 04:07:27.137694 22078 consensus_queue.cc:260] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } }
I20260924 04:07:27.137844 22078 raft_consensus.cc:388] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:27.137941 22078 raft_consensus.cc:482] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:27.138036 22078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:27.138952 22078 raft_consensus.cc:504] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } }
I20260924 04:07:27.139299 22078 leader_election.cc:302] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b98c0ae22a7f4ce3bb3c9da3753ab51c; no voters:
I20260924 04:07:27.139590 22078 leader_election.cc:288] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:27.139701 22083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:27.139868 22083 raft_consensus.cc:686] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [term 1 LEADER]: Becoming Leader. State: Replica: b98c0ae22a7f4ce3bb3c9da3753ab51c, State: Running, Role: LEADER
I20260924 04:07:27.140265 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:27.126813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44497 (local address 127.0.98.62:39617)
0924 04:07:27.127211 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:27.127227 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:27.127264 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:07:27.127947 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:27.127971 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:27.128358 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:27.128666 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:27.129583 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:27.130603 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:27.131773 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:27.132335 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:27.132485 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:27.134866 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:27.134871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:27.134884 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:27.134911 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:27.137083 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:27.137654 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:27.137658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:27.137663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:27.137775 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:27.138105 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:27.138109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:27.138111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:27.138502 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:27.138701 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:27.139034 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 04:07:27.139276 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260924 04:07:27.140244 22083 consensus_queue.cc:237] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } }
I20260924 04:07:27.140915 22078 sys_catalog.cc:564] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:27.142002 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } } }
I20260924 04:07:27.142125 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:27.142343 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98c0ae22a7f4ce3bb3c9da3753ab51c. Latest consensus state: current_term: 1 leader_uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98c0ae22a7f4ce3bb3c9da3753ab51c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39617 } } }
I20260924 04:07:27.142434 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:27.142705 22092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:27.143255 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:27.146178 22092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:27.152009 22092 catalog_manager.cc:1347] Generated new cluster ID: 50dd2eb2e6244df887f1a31daa912c64
I20260924 04:07:27.152083 22092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:27.159471 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:27.145644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33477 (local address 127.0.98.62:39617)
0924 04:07:27.145742 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:27.145746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:27.145965 (+ 219us) server_negotiation.cc:407] Connection header received
0924 04:07:27.146089 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:27.146093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:27.146140 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:27.146214 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:27.147018 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:27.147821 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:27.148955 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:27.149305 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:27.149410 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:27.150961 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:27.150965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:27.150968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:27.150994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:27.156591 (+ 5597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:27.157404 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:27.157408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:27.157409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:27.157511 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:27.157933 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:27.157936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:27.157938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:27.158171 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:27.158360 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:27.159151 (+ 791us) server_negotiation.cc:299] Negotiation successful
0924 04:07:27.159264 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:07:27.171505 22092 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:27.172866 22092 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:27.177377 22092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b98c0ae22a7f4ce3bb3c9da3753ab51c: Generated new TSK 0
I20260924 04:07:27.178299 22092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:27.187633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22010
W20260924 04:07:27.192301 21995 connection.cc:535] client connection to 127.0.98.62:39617 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39617): Connection reset by peer (error 104)
2026-09-24T04:07:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:27 dist-test-slave-zcsk krb5kdc[22109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:27 dist-test-slave-zcsk krb5kdc[22109](info): set up 2 sockets
Sep 24 04:07:27 dist-test-slave-zcsk krb5kdc[22109](info): commencing operation
krb5kdc: starting...
W20260924 04:07:28.139930 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:28 dist-test-slave-zcsk krb5kdc[22109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:28.259058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:28.369252 22122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:28.369568 22122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:28.369633 22122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:28.373775 22122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:28.373854 22122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:28.373883 22122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:28.373904 22122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:28.373924 22122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:28.379377 22122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:28.380483 22122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:28.381494 22122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:28.389770 22122 server_base.cc:1034] running on GCE node
W20260924 04:07:28.389870 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:28.389740 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:28.389747 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:28.390519 22122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:28.391577 22122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:28.392765 22122 hybrid_clock.cc:648] HybridClock initialized: now 1790222848392747 us; error 28 us; skew 500 ppm
Sep 24 04:07:28 dist-test-slave-zcsk krb5kdc[22109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:28.395498 22122 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:28.396373 22122 webserver.cc:460] Webserver started at http://127.0.98.62:43389/ using document root <none> and password file <none>
I20260924 04:07:28.396938 22122 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:28.397006 22122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:28.397207 22122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:28.399197 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "87d2dfc7889243a7b80f15c1d68bb771"
format_stamp: "Formatted at 2026-09-24 04:07:28 on dist-test-slave-zcsk"
server_key: "2f2e8de6f1044e55ad1e7d3a039c7f5c"
server_key_iv: "9f6bb3b4e92d4e3e4a69434229db2565"
server_key_version: "encryptionkey@0"
I20260924 04:07:28.399729 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "87d2dfc7889243a7b80f15c1d68bb771"
format_stamp: "Formatted at 2026-09-24 04:07:28 on dist-test-slave-zcsk"
server_key: "2f2e8de6f1044e55ad1e7d3a039c7f5c"
server_key_iv: "9f6bb3b4e92d4e3e4a69434229db2565"
server_key_version: "encryptionkey@0"
I20260924 04:07:28.403345 22122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:07:28.405741 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:28.406939 22122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:07:28.407048 22122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "87d2dfc7889243a7b80f15c1d68bb771"
format_stamp: "Formatted at 2026-09-24 04:07:28 on dist-test-slave-zcsk"
server_key: "2f2e8de6f1044e55ad1e7d3a039c7f5c"
server_key_iv: "9f6bb3b4e92d4e3e4a69434229db2565"
server_key_version: "encryptionkey@0"
I20260924 04:07:28.407142 22122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:28.420341 22122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:28.423229 22122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:28.423413 22122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:28.430377 22122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40863
I20260924 04:07:28.430389 22189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40863 every 8 connection(s)
I20260924 04:07:28.431337 22122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:28.434124 22190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:28.435369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22122
I20260924 04:07:28.435508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:28.435781 392 external_mini_cluster.cc:1428] Setting key 0504a7ccdb2e647f8734571029b65576
I20260924 04:07:28.440501 22190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771: Bootstrap starting.
I20260924 04:07:28.443099 22190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:28.443979 22190 log.cc:824] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:28.446041 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:28.437207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59701 (local address 127.0.98.62:40863)
0924 04:07:28.437638 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:28.437658 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:28.437700 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:07:28.438489 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:28.438515 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:28.438963 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:28.439337 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:28.439764 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:28.441184 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:28.442198 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:28.442881 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:28.443036 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:28.444052 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:28.444058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:28.444075 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:28.444561 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:28.444620 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:28.444643 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:28.444747 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:28.444964 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 04:07:28.445103 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260924 04:07:28.446272 22190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771: No bootstrap required, opened a new log
I20260924 04:07:28.449213 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:28.450569 22190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d2dfc7889243a7b80f15c1d68bb771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } }
I20260924 04:07:28.450892 22190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:28.450953 22190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d2dfc7889243a7b80f15c1d68bb771, State: Initialized, Role: FOLLOWER
I20260924 04:07:28.451448 22190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d2dfc7889243a7b80f15c1d68bb771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } }
I20260924 04:07:28.451673 22190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:28.451745 22190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:28.451839 22190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:28.452800 22190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d2dfc7889243a7b80f15c1d68bb771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } }
I20260924 04:07:28.453154 22190 leader_election.cc:302] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d2dfc7889243a7b80f15c1d68bb771; no voters:
I20260924 04:07:28.453456 22190 leader_election.cc:288] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:28.453543 22200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:28.453730 22200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [term 1 LEADER]: Becoming Leader. State: Replica: 87d2dfc7889243a7b80f15c1d68bb771, State: Running, Role: LEADER
I20260924 04:07:28.454097 22200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d2dfc7889243a7b80f15c1d68bb771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } }
I20260924 04:07:28.454839 22190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:28.455806 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d2dfc7889243a7b80f15c1d68bb771. Latest consensus state: current_term: 1 leader_uuid: "87d2dfc7889243a7b80f15c1d68bb771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d2dfc7889243a7b80f15c1d68bb771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } } }
I20260924 04:07:28.455951 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:28.456143 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:28.451493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52579 (local address 127.0.98.62:40863)
0924 04:07:28.451629 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:28.451634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:28.451828 (+ 194us) server_negotiation.cc:407] Connection header received
0924 04:07:28.452080 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:28.452085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:28.452130 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:28.452200 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:28.452505 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:28.453109 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:28.454146 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:28.454449 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:28.454546 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:28.455492 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:28.455497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:28.455500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:28.455516 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:28.455547 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:28.455564 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:28.455633 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:28.455948 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 04:07:28.455992 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:07:28.456418 22202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87d2dfc7889243a7b80f15c1d68bb771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d2dfc7889243a7b80f15c1d68bb771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40863 } } }
I20260924 04:07:28.456609 22202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:28.456911 22209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:28.459679 22209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:28.465270 22209 catalog_manager.cc:1347] Generated new cluster ID: 1790f16146d141b2898ca4dd3f4a2956
I20260924 04:07:28.465350 22209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:28.501997 22209 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:28.502967 22209 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:28.509320 22209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87d2dfc7889243a7b80f15c1d68bb771: Generated new TSK 0
I20260924 04:07:28.510039 22209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:28.522254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22122
W20260924 04:07:28.526821 22107 connection.cc:535] client connection to 127.0.98.62:40863 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40863): Connection reset by peer (error 104)
2026-09-24T04:07:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:28 dist-test-slave-zcsk krb5kdc[22221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:28 dist-test-slave-zcsk krb5kdc[22221](info): set up 2 sockets
Sep 24 04:07:28 dist-test-slave-zcsk krb5kdc[22221](info): commencing operation
krb5kdc: starting...
W20260924 04:07:29.475971 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:29 dist-test-slave-zcsk krb5kdc[22221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:29.595690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:29.705791 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:29.706120 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:29.706167 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:29.710323 22234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:29.710397 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:29.710417 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:29.710433 22234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:29.710445 22234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:29.716099 22234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:29.717247 22234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:29.718287 22234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:29.726734 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:29.726742 22239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:29.726748 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:29.727156 22234 server_base.cc:1034] running on GCE node
I20260924 04:07:29.727739 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:29.728924 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:29.730118 22234 hybrid_clock.cc:648] HybridClock initialized: now 1790222849730087 us; error 43 us; skew 500 ppm
Sep 24 04:07:29 dist-test-slave-zcsk krb5kdc[22221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:29.733013 22234 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:29.733944 22234 webserver.cc:460] Webserver started at http://127.0.98.62:46747/ using document root <none> and password file <none>
I20260924 04:07:29.734561 22234 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:29.734642 22234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:29.734863 22234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:29.737046 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "5b18ca47b73d43d4b8dcf25f43df2072"
format_stamp: "Formatted at 2026-09-24 04:07:29 on dist-test-slave-zcsk"
server_key: "6941d7bbec0bac55a2b04263cb0e2975"
server_key_iv: "e2e2f1f7a7424f0bd3ee69a6ebc16bc4"
server_key_version: "encryptionkey@0"
I20260924 04:07:29.737622 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b18ca47b73d43d4b8dcf25f43df2072"
format_stamp: "Formatted at 2026-09-24 04:07:29 on dist-test-slave-zcsk"
server_key: "6941d7bbec0bac55a2b04263cb0e2975"
server_key_iv: "e2e2f1f7a7424f0bd3ee69a6ebc16bc4"
server_key_version: "encryptionkey@0"
I20260924 04:07:29.741484 22234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:07:29.743991 22249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:29.745201 22234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:07:29.745316 22234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "5b18ca47b73d43d4b8dcf25f43df2072"
format_stamp: "Formatted at 2026-09-24 04:07:29 on dist-test-slave-zcsk"
server_key: "6941d7bbec0bac55a2b04263cb0e2975"
server_key_iv: "e2e2f1f7a7424f0bd3ee69a6ebc16bc4"
server_key_version: "encryptionkey@0"
I20260924 04:07:29.745412 22234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:29.749876 22234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:29.752935 22234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:29.753103 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:29.759829 22234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42155
I20260924 04:07:29.759855 22301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42155 every 8 connection(s)
I20260924 04:07:29.760840 22234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:29.761848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22234
I20260924 04:07:29.761986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:29.762327 392 external_mini_cluster.cc:1428] Setting key 436bfd91c621867f889a6849e124035f
I20260924 04:07:29.764387 22302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:29.768326 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:29.763813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56875 (local address 127.0.98.62:42155)
0924 04:07:29.764280 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:29.764301 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:29.764343 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:07:29.765204 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:29.765228 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:29.765623 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:29.766003 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:29.766812 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:29.766818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:29.766834 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:29.766846 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:29.766910 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:29.766940 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:29.767090 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:29.767172 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 04:07:29.767335 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260924 04:07:29.771377 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:29.771500 22302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072: Bootstrap starting.
I20260924 04:07:29.773829 22302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:29.774573 22302 log.cc:824] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:29.775256 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:29.773594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54693 (local address 127.0.98.62:42155)
0924 04:07:29.773722 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:29.773727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:29.773878 (+ 151us) server_negotiation.cc:407] Connection header received
0924 04:07:29.774025 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:29.774030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:29.774066 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:29.774137 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:29.774857 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:29.774861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:29.774864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:29.774877 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:29.774905 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:29.774918 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:29.774988 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:29.775082 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:07:29.775124 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:07:29.776818 22302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072: No bootstrap required, opened a new log
I20260924 04:07:29.780833 22302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b18ca47b73d43d4b8dcf25f43df2072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42155 } }
I20260924 04:07:29.781096 22302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:29.781170 22302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b18ca47b73d43d4b8dcf25f43df2072, State: Initialized, Role: FOLLOWER
I20260924 04:07:29.781630 22302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b18ca47b73d43d4b8dcf25f43df2072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42155 } }
I20260924 04:07:29.781816 22302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:29.781886 22302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:29.781996 22302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:29.782995 22302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b18ca47b73d43d4b8dcf25f43df2072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42155 } }
I20260924 04:07:29.783344 22302 leader_election.cc:302] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b18ca47b73d43d4b8dcf25f43df2072; no voters:
I20260924 04:07:29.783689 22302 leader_election.cc:288] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:29.783823 22312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:29.784044 22312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [term 1 LEADER]: Becoming Leader. State: Replica: 5b18ca47b73d43d4b8dcf25f43df2072, State: Running, Role: LEADER
I20260924 04:07:29.784353 22312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b18ca47b73d43d4b8dcf25f43df2072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42155 } }
I20260924 04:07:29.785000 22302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:29.785933 22314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b18ca47b73d43d4b8dcf25f43df2072. Latest consensus state: current_term: 1 leader_uuid: "5b18ca47b73d43d4b8dcf25f43df2072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b18ca47b73d43d4b8dcf25f43df2072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42155 } } }
I20260924 04:07:29.786077 22314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:29.786342 22313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b18ca47b73d43d4b8dcf25f43df2072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b18ca47b73d43d4b8dcf25f43df2072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42155 } } }
I20260924 04:07:29.786433 22313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:29.786446 22321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:29.789402 22321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:29.795126 22321 catalog_manager.cc:1347] Generated new cluster ID: cf39519ae0e74f06ad6e230ed4eb8eb9
I20260924 04:07:29.795195 22321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:29.846575 22321 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:29.846911 22321 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:29.852687 22321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b18ca47b73d43d4b8dcf25f43df2072: Generated new TSK 0
I20260924 04:07:29.853380 22321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:29.861728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22234
2026-09-24T04:07:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:29 dist-test-slave-zcsk krb5kdc[22333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:29 dist-test-slave-zcsk krb5kdc[22333](info): set up 2 sockets
Sep 24 04:07:29 dist-test-slave-zcsk krb5kdc[22333](info): commencing operation
krb5kdc: starting...
W20260924 04:07:30.820015 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:30 dist-test-slave-zcsk krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:30.943421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34745
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:31.056572 22346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:31.056885 22346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:31.056947 22346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:31.061054 22346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:31.061126 22346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:31.061161 22346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:31.061185 22346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:31.061211 22346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:31.066852 22346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:31.068058 22346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:31.069144 22346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:31.077991 22351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:31.077991 22352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:31.078217 22346 server_base.cc:1034] running on GCE node
W20260924 04:07:31.078014 22354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:31.079069 22346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:31.080340 22346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:31.081516 22346 hybrid_clock.cc:648] HybridClock initialized: now 1790222851081486 us; error 45 us; skew 500 ppm
Sep 24 04:07:31 dist-test-slave-zcsk krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:31.090366 22346 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:31.091423 22346 webserver.cc:460] Webserver started at http://127.0.98.62:42667/ using document root <none> and password file <none>
I20260924 04:07:31.092046 22346 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:31.092120 22346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:31.092342 22346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:31.094480 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a"
format_stamp: "Formatted at 2026-09-24 04:07:31 on dist-test-slave-zcsk"
server_key: "1714fc57a34e909b6ea1e51a519a4065"
server_key_iv: "b8d65c86a594fcf878895f91655904c8"
server_key_version: "encryptionkey@0"
I20260924 04:07:31.095067 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a"
format_stamp: "Formatted at 2026-09-24 04:07:31 on dist-test-slave-zcsk"
server_key: "1714fc57a34e909b6ea1e51a519a4065"
server_key_iv: "b8d65c86a594fcf878895f91655904c8"
server_key_version: "encryptionkey@0"
I20260924 04:07:31.099093 22346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:07:31.101924 22361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:31.103302 22346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:07:31.103446 22346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a"
format_stamp: "Formatted at 2026-09-24 04:07:31 on dist-test-slave-zcsk"
server_key: "1714fc57a34e909b6ea1e51a519a4065"
server_key_iv: "b8d65c86a594fcf878895f91655904c8"
server_key_version: "encryptionkey@0"
I20260924 04:07:31.103567 22346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:31.119009 22346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:31.122817 22346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:31.123023 22346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:31.131018 22346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34745
I20260924 04:07:31.131080 22413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34745 every 8 connection(s)
I20260924 04:07:31.132107 22346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:31.135072 22414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:31.140197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22346
I20260924 04:07:31.140374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:31.140640 392 external_mini_cluster.cc:1428] Setting key 3d3ed67d8964bab1448bcf307bb06a4f
I20260924 04:07:31.141111 22414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a: Bootstrap starting.
I20260924 04:07:31.143594 22414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:31.144550 22414 log.cc:824] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:31.146747 22414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a: No bootstrap required, opened a new log
Sep 24 04:07:31 dist-test-slave-zcsk krb5kdc[22333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:31.151326 22414 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260924 04:07:31.151615 22414 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:31.151676 22414 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7c2ea3dfb043e4bc58b768a0f77f6a, State: Initialized, Role: FOLLOWER
I20260924 04:07:31.152201 22414 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260924 04:07:31.152341 22414 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:31.152406 22414 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:31.152498 22414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:31.153613 22414 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260924 04:07:31.153980 22414 leader_election.cc:302] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7c2ea3dfb043e4bc58b768a0f77f6a; no voters:
I20260924 04:07:31.154289 22414 leader_election.cc:288] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:31.154438 22419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:31.154672 22419 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [term 1 LEADER]: Becoming Leader. State: Replica: ca7c2ea3dfb043e4bc58b768a0f77f6a, State: Running, Role: LEADER
I20260924 04:07:31.155347 22419 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260924 04:07:31.155633 22414 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:31.156457 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:31.142077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33955 (local address 127.0.98.62:34745)
0924 04:07:31.142534 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:31.142550 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:31.142586 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:07:31.143281 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:31.143305 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:31.143660 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:31.144025 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:31.144896 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:31.146091 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:31.147243 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:31.147730 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:31.147876 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:31.150749 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:31.150756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:31.150773 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:31.150805 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:31.152740 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:31.153251 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:31.153256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:31.153262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:31.153423 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:31.153733 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:31.153736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:31.153738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:31.154230 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:31.154506 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:31.155008 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 04:07:31.155413 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260924 04:07:31.156940 22421 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7c2ea3dfb043e4bc58b768a0f77f6a. Latest consensus state: current_term: 1 leader_uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } } }
I20260924 04:07:31.157060 22421 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:31.158320 22420 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7c2ea3dfb043e4bc58b768a0f77f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } } }
I20260924 04:07:31.158418 22420 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:31.160058 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 04:07:31.160574 22434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:31.160651 22434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:31.160717 22430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:31.161512 22430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:31.167811 22430 catalog_manager.cc:1347] Generated new cluster ID: cecea5b228414d268b23f9141ae004da
I20260924 04:07:31.167876 22430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:31.171207 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:31.162111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47119 (local address 127.0.98.62:34745)
0924 04:07:31.162215 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:31.162220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:31.162477 (+ 257us) server_negotiation.cc:407] Connection header received
0924 04:07:31.162607 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:31.162610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:31.162660 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:31.162738 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:31.163544 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:31.164131 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:31.165119 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:31.165409 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:31.165512 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:31.167298 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:31.167302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:31.167306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:31.167335 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:31.169211 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:31.169751 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:31.169755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:31.169757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:31.169872 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:31.170209 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:31.170212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:31.170214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:31.170443 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:31.170545 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:31.170923 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 04:07:31.171045 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:07:31.186340 22430 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:31.187309 22430 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:31.195252 22430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7c2ea3dfb043e4bc58b768a0f77f6a: Generated new TSK 0
I20260924 04:07:31.196022 22430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:31.200029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22346
2026-09-24T04:07:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:31 dist-test-slave-zcsk krb5kdc[22445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:31 dist-test-slave-zcsk krb5kdc[22445](info): set up 2 sockets
Sep 24 04:07:31 dist-test-slave-zcsk krb5kdc[22445](info): commencing operation
krb5kdc: starting...
W20260924 04:07:32.176355 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:32 dist-test-slave-zcsk krb5kdc[22445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:32.294598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40971
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:32.404600 22458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:32.404896 22458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:32.404958 22458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:32.409029 22458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:32.409102 22458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:32.409128 22458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:32.409152 22458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:32.409178 22458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:32.414637 22458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:32.415747 22458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:32.416869 22458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:32.425351 22463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:32.425448 22458 server_base.cc:1034] running on GCE node
W20260924 04:07:32.425460 22466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:32.425367 22464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:32.426198 22458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:32.427356 22458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:32.428547 22458 hybrid_clock.cc:648] HybridClock initialized: now 1790222852428518 us; error 44 us; skew 500 ppm
Sep 24 04:07:32 dist-test-slave-zcsk krb5kdc[22445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:32.431604 22458 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:32.432613 22458 webserver.cc:460] Webserver started at http://127.0.98.62:43223/ using document root <none> and password file <none>
I20260924 04:07:32.433167 22458 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:32.433235 22458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:32.433462 22458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:32.435590 22458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "3bfa376cf8f14423b5c511518e9dd5f9"
format_stamp: "Formatted at 2026-09-24 04:07:32 on dist-test-slave-zcsk"
server_key: "d5e8be1a50ec0f914840a0e5a24c0b8b"
server_key_iv: "86f6565d47a3898c2fe3e26558f4232b"
server_key_version: "encryptionkey@0"
I20260924 04:07:32.436141 22458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bfa376cf8f14423b5c511518e9dd5f9"
format_stamp: "Formatted at 2026-09-24 04:07:32 on dist-test-slave-zcsk"
server_key: "d5e8be1a50ec0f914840a0e5a24c0b8b"
server_key_iv: "86f6565d47a3898c2fe3e26558f4232b"
server_key_version: "encryptionkey@0"
I20260924 04:07:32.439874 22458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:07:32.442505 22473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:32.444036 22458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:07:32.444164 22458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "3bfa376cf8f14423b5c511518e9dd5f9"
format_stamp: "Formatted at 2026-09-24 04:07:32 on dist-test-slave-zcsk"
server_key: "d5e8be1a50ec0f914840a0e5a24c0b8b"
server_key_iv: "86f6565d47a3898c2fe3e26558f4232b"
server_key_version: "encryptionkey@0"
I20260924 04:07:32.444283 22458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:32.468286 22458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:32.471582 22458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:32.471767 22458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:32.478849 22458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40971
I20260924 04:07:32.478861 22525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40971 every 8 connection(s)
I20260924 04:07:32.479809 22458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:32.480823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22458
I20260924 04:07:32.480926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:32.481208 392 external_mini_cluster.cc:1428] Setting key ffc294307ac625bb626a8acf886621a1
I20260924 04:07:32.482930 22526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:32.489751 22526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9: Bootstrap starting.
I20260924 04:07:32.491400 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:32.482643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35033 (local address 127.0.98.62:40971)
0924 04:07:32.483070 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:32.483089 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:32.483131 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:07:32.483993 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:32.484016 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:32.484473 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:32.484855 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:32.485264 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:32.486744 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:32.487693 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:32.488403 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:32.488578 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:32.489393 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:32.489399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:32.489415 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:32.489910 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:32.489978 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:32.490007 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:32.490102 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:32.490238 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 04:07:32.490445 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260924 04:07:32.492336 22526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:32.493112 22526 log.cc:824] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:32.494324 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:32.495180 22526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9: No bootstrap required, opened a new log
I20260924 04:07:32.498351 22526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfa376cf8f14423b5c511518e9dd5f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } }
I20260924 04:07:32.498581 22526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:32.498648 22526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfa376cf8f14423b5c511518e9dd5f9, State: Initialized, Role: FOLLOWER
I20260924 04:07:32.499133 22526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfa376cf8f14423b5c511518e9dd5f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } }
I20260924 04:07:32.499300 22526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:32.499368 22526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:32.499465 22526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:32.500353 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:32.496220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42975 (local address 127.0.98.62:40971)
0924 04:07:32.496319 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:32.496324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:32.496503 (+ 179us) server_negotiation.cc:407] Connection header received
0924 04:07:32.496626 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:32.496629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:32.496668 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:32.496727 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:32.496981 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:32.497944 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:32.498745 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:32.499077 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:32.499181 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:32.499980 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:32.499983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:32.499986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:32.499999 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:32.500026 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:32.500039 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:32.500114 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:32.500173 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:07:32.500225 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:07:32.500568 22526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfa376cf8f14423b5c511518e9dd5f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } }
I20260924 04:07:32.500914 22526 leader_election.cc:302] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bfa376cf8f14423b5c511518e9dd5f9; no voters:
I20260924 04:07:32.501263 22526 leader_election.cc:288] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:32.501369 22536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:32.501570 22536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 3bfa376cf8f14423b5c511518e9dd5f9, State: Running, Role: LEADER
I20260924 04:07:32.501879 22536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfa376cf8f14423b5c511518e9dd5f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } }
I20260924 04:07:32.502420 22526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:32.503537 22538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfa376cf8f14423b5c511518e9dd5f9. Latest consensus state: current_term: 1 leader_uuid: "3bfa376cf8f14423b5c511518e9dd5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfa376cf8f14423b5c511518e9dd5f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } } }
I20260924 04:07:32.503548 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bfa376cf8f14423b5c511518e9dd5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfa376cf8f14423b5c511518e9dd5f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40971 } } }
I20260924 04:07:32.503670 22538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:32.503675 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:32.504098 22545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:32.507354 22545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:32.513110 22545 catalog_manager.cc:1347] Generated new cluster ID: 22caf2d9a1134d56875ca06609f20b8e
I20260924 04:07:32.513192 22545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:32.525936 22545 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:32.527238 22545 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:32.538290 22545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bfa376cf8f14423b5c511518e9dd5f9: Generated new TSK 0
I20260924 04:07:32.539183 22545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:32.547451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22458
2026-09-24T04:07:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:32 dist-test-slave-zcsk krb5kdc[22557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:32 dist-test-slave-zcsk krb5kdc[22557](info): set up 2 sockets
Sep 24 04:07:32 dist-test-slave-zcsk krb5kdc[22557](info): commencing operation
krb5kdc: starting...
W20260924 04:07:33.499859 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:33 dist-test-slave-zcsk krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:33.619007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:33.727819 22570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:33.728142 22570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:33.728185 22570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:33.732160 22570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:33.732220 22570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:33.732237 22570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:33.732252 22570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:33.732266 22570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:33.737527 22570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:33.738580 22570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:33.739542 22570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:33.747826 22576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:33.747943 22570 server_base.cc:1034] running on GCE node
W20260924 04:07:33.747813 22575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:33.747807 22578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:33.748612 22570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:33.749715 22570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:33.750869 22570 hybrid_clock.cc:648] HybridClock initialized: now 1790222853750849 us; error 33 us; skew 500 ppm
Sep 24 04:07:33 dist-test-slave-zcsk krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:33.753546 22570 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:33.754478 22570 webserver.cc:460] Webserver started at http://127.0.98.62:41205/ using document root <none> and password file <none>
I20260924 04:07:33.755012 22570 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:33.755079 22570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:33.755270 22570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:33.757346 22570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "e668f3dcb7d8492db509ee0e28de8ee4"
format_stamp: "Formatted at 2026-09-24 04:07:33 on dist-test-slave-zcsk"
server_key: "2bedb2b4a378ee542d7e0fdf6e664809"
server_key_iv: "6b922c56470cb6e35f3bb658f96c2559"
server_key_version: "encryptionkey@0"
I20260924 04:07:33.757884 22570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "e668f3dcb7d8492db509ee0e28de8ee4"
format_stamp: "Formatted at 2026-09-24 04:07:33 on dist-test-slave-zcsk"
server_key: "2bedb2b4a378ee542d7e0fdf6e664809"
server_key_iv: "6b922c56470cb6e35f3bb658f96c2559"
server_key_version: "encryptionkey@0"
I20260924 04:07:33.761561 22570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 04:07:33.764182 22585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:33.765508 22570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:07:33.765616 22570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "e668f3dcb7d8492db509ee0e28de8ee4"
format_stamp: "Formatted at 2026-09-24 04:07:33 on dist-test-slave-zcsk"
server_key: "2bedb2b4a378ee542d7e0fdf6e664809"
server_key_iv: "6b922c56470cb6e35f3bb658f96c2559"
server_key_version: "encryptionkey@0"
I20260924 04:07:33.765750 22570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:33.770416 22570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:33.773527 22570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:33.773705 22570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:33.780504 22570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38575
I20260924 04:07:33.780586 22637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38575 every 8 connection(s)
I20260924 04:07:33.781436 22570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:33.784313 22638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:33.784680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22570
I20260924 04:07:33.784788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:33.785034 392 external_mini_cluster.cc:1428] Setting key 01c7989e8952c47e075425f5444c6223
I20260924 04:07:33.790570 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:33.786285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57633 (local address 127.0.98.62:38575)
0924 04:07:33.786688 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:33.786708 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:33.786752 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:07:33.787542 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:33.787566 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:33.787983 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:33.788370 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:33.789063 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:33.789069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:33.789085 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:33.789098 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:33.789160 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:33.789188 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:33.789339 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:33.789411 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:07:33.789578 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":91,"threads_started":1}
I20260924 04:07:33.791647 22638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4: Bootstrap starting.
I20260924 04:07:33.793336 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:33.794248 22638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:33.795183 22638 log.cc:824] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:33.797340 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:33.795684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35861 (local address 127.0.98.62:38575)
0924 04:07:33.795800 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:33.795805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:33.796051 (+ 246us) server_negotiation.cc:407] Connection header received
0924 04:07:33.796195 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:33.796197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:33.796241 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:33.796302 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:33.796977 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:33.796980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:33.796981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:07:33.796990 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:33.797013 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:07:33.797023 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:33.797083 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:33.797182 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 04:07:33.797225 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:07:33.797576 22638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4: No bootstrap required, opened a new log
I20260924 04:07:33.801303 22638 raft_consensus.cc:348] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e668f3dcb7d8492db509ee0e28de8ee4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38575 } }
I20260924 04:07:33.801578 22638 raft_consensus.cc:374] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:07:33.801628 22638 raft_consensus.cc:729] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e668f3dcb7d8492db509ee0e28de8ee4, State: Initialized, Role: FOLLOWER
I20260924 04:07:33.802044 22638 consensus_queue.cc:260] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e668f3dcb7d8492db509ee0e28de8ee4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38575 } }
I20260924 04:07:33.802175 22638 raft_consensus.cc:388] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:07:33.802227 22638 raft_consensus.cc:482] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:07:33.802325 22638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:33.803447 22638 raft_consensus.cc:504] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e668f3dcb7d8492db509ee0e28de8ee4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38575 } }
I20260924 04:07:33.803764 22638 leader_election.cc:302] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e668f3dcb7d8492db509ee0e28de8ee4; no voters:
I20260924 04:07:33.804113 22638 leader_election.cc:288] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:07:33.804175 22648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:33.804391 22648 raft_consensus.cc:686] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [term 1 LEADER]: Becoming Leader. State: Replica: e668f3dcb7d8492db509ee0e28de8ee4, State: Running, Role: LEADER
I20260924 04:07:33.804715 22648 consensus_queue.cc:237] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e668f3dcb7d8492db509ee0e28de8ee4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38575 } }
I20260924 04:07:33.805270 22638 sys_catalog.cc:564] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:33.806237 22649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e668f3dcb7d8492db509ee0e28de8ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e668f3dcb7d8492db509ee0e28de8ee4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38575 } } }
I20260924 04:07:33.806372 22649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:33.806670 22650 sys_catalog.cc:455] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e668f3dcb7d8492db509ee0e28de8ee4. Latest consensus state: current_term: 1 leader_uuid: "e668f3dcb7d8492db509ee0e28de8ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e668f3dcb7d8492db509ee0e28de8ee4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38575 } } }
I20260924 04:07:33.806758 22657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:33.806845 22650 sys_catalog.cc:458] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:33.810073 22657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:33.815603 22657 catalog_manager.cc:1347] Generated new cluster ID: e256b4f165474f648cbfe95778917895
I20260924 04:07:33.815688 22657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:33.824970 22657 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:33.825390 22657 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:33.834515 22657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e668f3dcb7d8492db509ee0e28de8ee4: Generated new TSK 0
I20260924 04:07:33.835227 22657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:07:33.844441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22570
2026-09-24T04:07:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1296 ms)
[----------] 18 tests from AuthTokenIssuingTest (23938 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:33 dist-test-slave-zcsk krb5kdc[22669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:33 dist-test-slave-zcsk krb5kdc[22669](info): set up 2 sockets
Sep 24 04:07:33 dist-test-slave-zcsk krb5kdc[22669](info): commencing operation
krb5kdc: starting...
W20260924 04:07:34.795938 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:34 dist-test-slave-zcsk krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:34.911088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37619
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37619,127.0.98.61:43867,127.0.98.60:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:35.020479 22682 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:07:35.020782 22682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:35.020818 22682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:35.020876 22682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:35.024962 22682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:35.025028 22682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:35.025054 22682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:35.025076 22682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:35.025104 22682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:35.030414 22682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37619,127.0.98.61:43867,127.0.98.60:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:35.031505 22682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:35.032563 22682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:35.041250 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:35.041244 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:35.041292 22682 server_base.cc:1034] running on GCE node
W20260924 04:07:35.041244 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:35.042038 22682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:35.043174 22682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:35.044334 22682 hybrid_clock.cc:648] HybridClock initialized: now 1790222855044318 us; error 30 us; skew 500 ppm
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:35.046990 22682 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:35.047951 22682 webserver.cc:460] Webserver started at http://127.0.98.62:33455/ using document root <none> and password file <none>
I20260924 04:07:35.048491 22682 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:35.048556 22682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:35.048763 22682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:35.050827 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "f258770cb82d47adb66ead1ebd6571a8"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "e1ce6f54a50ecdfa50477f0eda959288"
server_key_iv: "ad1101d081036b7e9b926fa0d44a2ee2"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.051370 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "f258770cb82d47adb66ead1ebd6571a8"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "e1ce6f54a50ecdfa50477f0eda959288"
server_key_iv: "ad1101d081036b7e9b926fa0d44a2ee2"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.055183 22682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:07:35.057782 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:35.059106 22682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:07:35.059252 22682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "f258770cb82d47adb66ead1ebd6571a8"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "e1ce6f54a50ecdfa50477f0eda959288"
server_key_iv: "ad1101d081036b7e9b926fa0d44a2ee2"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.059415 22682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:35.069303 22682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:35.072250 22682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:35.072445 22682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:35.079855 22682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37619
I20260924 04:07:35.080166 22749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37619 every 8 connection(s)
I20260924 04:07:35.080931 22682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:35.083851 22750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:35.087139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22682
I20260924 04:07:35.087270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:35.087543 392 external_mini_cluster.cc:1428] Setting key cbe4457e8f24e7d07a6d5524f0bfb8a2
I20260924 04:07:35.088160 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:35.104871 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.091567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.98.62:37619)
0924 04:07:35.091815 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.091827 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.091870 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:07:35.092615 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.092621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.092877 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.093081 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.093916 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.095312 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.096411 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.096998 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.097167 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.100398 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.100404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.100415 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.100436 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.101890 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.102324 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.102329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.102330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.102432 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.102706 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.102708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.102709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.103108 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.103224 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.103582 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.103798 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260924 04:07:35.104871 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.090810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37619 (local address 127.0.0.1:49800)
0924 04:07:35.091351 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.091398 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.092309 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.093109 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.093116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.093510 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.093767 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.093780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.095470 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.095473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.096291 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.096301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.097136 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.097138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.097235 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.097765 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.097782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.100236 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.101989 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.101997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.102008 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.102213 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.102511 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.102514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.102516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.102619 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.103209 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.103215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.103348 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.103493 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.103863 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":93,"thread_start_us":121,"threads_started":1}
I20260924 04:07:35.107426 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:35.108153 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.107896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38644)
0924 04:07:35.108016 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.108089 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:07:35.108356 22699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
I20260924 04:07:35.109154 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.108946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38654)
0924 04:07:35.109048 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.109089 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:07:35.109308 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43867: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:07:35.112906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43867
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37619,127.0.98.61:43867,127.0.98.60:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:07:35.158586 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } attempt: 1
I20260924 04:07:35.159564 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.159187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38664)
0924 04:07:35.159401 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.159470 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 04:07:35.160301 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.160074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38666)
0924 04:07:35.160185 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.160231 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:07:35.160416 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43867: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
W20260924 04:07:35.223577 22757 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:07:35.223948 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:35.223991 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:35.224056 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:35.228103 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:35.228168 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:35.228193 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:35.228214 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:35.228235 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:35.228555 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } attempt: 2
I20260924 04:07:35.229317 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.229042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38682)
0924 04:07:35.229182 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.229243 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:07:35.229943 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.229744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38692)
0924 04:07:35.229840 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.229885 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:07:35.230052 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43867: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43867: connect: Connection refused (error 111)
I20260924 04:07:35.233714 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37619,127.0.98.61:43867,127.0.98.60:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:35.234812 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:35.235795 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:35.243860 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:35.243860 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:35.243889 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:35.244006 22757 server_base.cc:1034] running on GCE node
I20260924 04:07:35.244702 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:35.245970 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:35.247166 22757 hybrid_clock.cc:648] HybridClock initialized: now 1790222855247147 us; error 36 us; skew 500 ppm
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:35.250002 22757 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:07:35.250918 22757 webserver.cc:460] Webserver started at http://127.0.98.61:38141/ using document root <none> and password file <none>
I20260924 04:07:35.251456 22757 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:35.251523 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:35.251726 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:35.253852 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/data/instance:
uuid: "03f8d8bd27c74fb8bbe0b28ec145f746"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "466a389db7476c0559f124339f497f3c"
server_key_iv: "fc2c39fb29bf7902d4612099b68346de"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.254415 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/wal/instance:
uuid: "03f8d8bd27c74fb8bbe0b28ec145f746"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "466a389db7476c0559f124339f497f3c"
server_key_iv: "fc2c39fb29bf7902d4612099b68346de"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.258298 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:07:35.260854 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:35.262135 22757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:07:35.262269 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/wal
uuid: "03f8d8bd27c74fb8bbe0b28ec145f746"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "466a389db7476c0559f124339f497f3c"
server_key_iv: "fc2c39fb29bf7902d4612099b68346de"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.262427 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:35.273414 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:35.276628 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:35.276795 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:35.283636 22757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43867
I20260924 04:07:35.283898 22824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43867 every 8 connection(s)
I20260924 04:07:35.284662 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:07:35.287423 22825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:35.288869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22757
I20260924 04:07:35.289012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-1/wal/instance
I20260924 04:07:35.289300 392 external_mini_cluster.cc:1428] Setting key 6c4012b79d6d462f73db0e19b5635516
I20260924 04:07:35.291898 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:35.306054 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.293937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.0.98.62:37619)
0924 04:07:35.294065 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.294068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.295133 (+ 1065us) server_negotiation.cc:407] Connection header received
0924 04:07:35.296108 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.296112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.296171 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.296250 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.297455 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.298037 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.299150 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.299486 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.299593 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.302263 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.302266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.302268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.302292 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.303774 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.304445 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.304447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.304449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.304527 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.304916 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.304918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.304920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.305086 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.305155 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.305824 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.305929 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:07:35.307265 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.294443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37619 (local address 127.0.0.1:49816)
0924 04:07:35.294978 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.295022 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.295850 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.296466 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.296473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.296804 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.297310 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.297323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.298157 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.298159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.299032 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.299043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.299665 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.299668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.299845 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.300487 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.300505 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.302091 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.303896 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.303903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.303943 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.304311 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.304620 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.304624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.304626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.304826 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.305181 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.305188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.305535 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.305786 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.306142 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":142,"threads_started":1}
I20260924 04:07:35.308677 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:07:35.315021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45501
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37619,127.0.98.61:43867,127.0.98.60:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:07:35.317509 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.309131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38706)
0924 04:07:35.309266 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.309280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.309354 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.309884 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.309893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.310063 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.310163 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.310168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.311100 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.311104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.311718 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.311727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.312475 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.312478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.312563 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.312975 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.312986 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.314253 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.315943 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.315946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.315948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.316163 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.316482 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.316485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.316487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.316570 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.317011 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.317013 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.317111 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.317217 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.317334 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 04:07:35.317566 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.309279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.98.61:43867)
0924 04:07:35.309557 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.309566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.309585 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:07:35.309619 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.309623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.309770 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.309877 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.310247 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.310976 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.311830 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.312350 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.312444 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.314372 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.314383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.314397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.314421 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.315820 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.316287 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.316293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.316294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.316385 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.316645 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.316647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.316649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.316914 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.317023 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.317318 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.317437 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":108,"threads_started":1}
I20260924 04:07:35.319689 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:35.320381 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.320105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38062)
0924 04:07:35.320226 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.320310 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:07:35.320598 22775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
I20260924 04:07:35.321197 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } attempt: 3
I20260924 04:07:35.321406 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.321183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38064)
0924 04:07:35.321293 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.321346 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:07:35.321802 22825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:07:35.329420 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.321823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.0.98.61:43867)
0924 04:07:35.321912 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.321916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.321928 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:07:35.321967 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.321970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.322009 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.322070 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.322471 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.322973 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.324242 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.324478 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.324559 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.326646 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.326649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.326650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.326671 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.327838 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.328347 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.328349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.328350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.328415 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.328691 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.328694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.328696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.328859 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.328938 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.329180 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.329280 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:07:35.329421 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.321579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38708)
0924 04:07:35.321690 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.321704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.321785 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.322114 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.322117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.322275 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.322372 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.322377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.323098 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.323101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.324131 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.324142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.324574 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.324577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.324680 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.325119 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.325130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.326525 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.328019 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.328023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.328025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.328267 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.328525 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.328528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.328530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.328609 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.328945 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.328948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.329039 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.329124 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.329257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:07:35.330475 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:35.331111 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.330899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38074)
0924 04:07:35.331010 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.331054 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:07:35.331753 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.331540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38078)
0924 04:07:35.331662 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.331705 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 04:07:35.331852 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
I20260924 04:07:35.362994 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } attempt: 1
I20260924 04:07:35.363817 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.363511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38094)
0924 04:07:35.363672 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.363735 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:07:35.364516 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.364296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38106)
0924 04:07:35.364392 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.364434 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:07:35.364629 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
I20260924 04:07:35.371071 22825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } attempt: 1
I20260924 04:07:35.371872 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.371563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38110)
0924 04:07:35.371713 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.371783 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:07:35.372551 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.372335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38124)
0924 04:07:35.372431 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.372490 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:07:35.372668 22825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
W20260924 04:07:35.427464 22831 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:07:35.427766 22831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:35.427788 22831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:35.427836 22831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:35.431975 22831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:35.432047 22831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:35.432066 22831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:35.432085 22831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:35.432099 22831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:35.437477 22831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37619,127.0.98.61:43867,127.0.98.60:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:35.438601 22831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:35.439620 22831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:07:35.439769 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } attempt: 2
I20260924 04:07:35.440636 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.440317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38138)
0924 04:07:35.440474 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.440560 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:07:35.440814 22825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } attempt: 2
I20260924 04:07:35.441319 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.441111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38144)
0924 04:07:35.441213 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.441259 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:07:35.441431 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
I20260924 04:07:35.441480 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.441178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38156)
0924 04:07:35.441286 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.441335 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 04:07:35.442065 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.441893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38172)
0924 04:07:35.441970 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.442008 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 04:07:35.442170 22825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45501: connect: Connection refused (error 111)
W20260924 04:07:35.448254 22840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:35.448247 22838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:35.448247 22837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:35.448432 22831 server_base.cc:1034] running on GCE node
I20260924 04:07:35.449061 22831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:35.450125 22831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:35.451318 22831 hybrid_clock.cc:648] HybridClock initialized: now 1790222855451292 us; error 37 us; skew 500 ppm
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:35.454376 22831 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:07:35.455407 22831 webserver.cc:460] Webserver started at http://127.0.98.60:43927/ using document root <none> and password file <none>
I20260924 04:07:35.456030 22831 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:35.456110 22831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:35.456326 22831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:35.458479 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/data/instance:
uuid: "c738370cb74848089204f723361aa790"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "76398196e7b8cbb22efd3636d4e81131"
server_key_iv: "9643c0060fa9fc4970c51fa8b6d23d06"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.459054 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/wal/instance:
uuid: "c738370cb74848089204f723361aa790"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "76398196e7b8cbb22efd3636d4e81131"
server_key_iv: "9643c0060fa9fc4970c51fa8b6d23d06"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.463001 22831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:07:35.465515 22847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:35.466763 22831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:07:35.466878 22831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/wal
uuid: "c738370cb74848089204f723361aa790"
format_stamp: "Formatted at 2026-09-24 04:07:35 on dist-test-slave-zcsk"
server_key: "76398196e7b8cbb22efd3636d4e81131"
server_key_iv: "9643c0060fa9fc4970c51fa8b6d23d06"
server_key_version: "encryptionkey@0"
I20260924 04:07:35.466980 22831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:35.496973 22831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:35.499862 22831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:35.500090 22831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:35.506775 22831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45501
I20260924 04:07:35.507016 22899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45501 every 8 connection(s)
I20260924 04:07:35.507758 22831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:07:35.510552 22900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:35.511449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 22831
I20260924 04:07:35.511545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222576786845-392-0/minicluster-data/master-2/wal/instance
I20260924 04:07:35.511788 392 external_mini_cluster.cc:1428] Setting key 5c13abbccd92e19804d71c1cfec23b1b
I20260924 04:07:35.514930 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:35.524915 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.516899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.0.98.62:37619)
0924 04:07:35.517028 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.517033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.517336 (+ 303us) server_negotiation.cc:407] Connection header received
0924 04:07:35.517463 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.517466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.517517 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.517604 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.518225 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.518701 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.519527 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.519738 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.519823 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.522086 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.522088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.522091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.522113 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.523317 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.523737 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.523740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.523741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.523810 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.524093 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.524096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.524098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.524294 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.524383 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.524712 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.524801 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:07:35.525400 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.513162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.0.98.60:45501)
0924 04:07:35.513582 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.513601 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.513645 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:07:35.514452 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.514475 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.514902 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.515266 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.515979 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.517203 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.518146 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.518623 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.518759 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.520974 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.520979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.520988 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.521009 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.522503 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.522911 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.522916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.522921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.523019 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.523272 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.523275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.523277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.523666 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.523834 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.524123 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.524394 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260924 04:07:35.525400 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.516814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37619 (local address 127.0.0.1:49820)
0924 04:07:35.517207 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.517256 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.517376 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.517653 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.517657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.517980 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.518098 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.518104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.518808 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.518811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.519417 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.519425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.519870 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.519874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.519990 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.520450 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.520465 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.521939 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.523417 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.523426 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.523431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.523647 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.523888 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.523891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.523893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.524007 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.524402 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.524408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.524515 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.524645 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.524801 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":134,"threads_started":1}
I20260924 04:07:35.526681 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:07:35.533321 22825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } attempt: 3
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:35.534979 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.527232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.0.98.61:43867)
0924 04:07:35.527333 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.527336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.527347 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:07:35.527383 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.527386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.527429 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.527509 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.528017 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.528558 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.529538 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.529865 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.529987 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.531826 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.531829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.531831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.531856 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.533333 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.533824 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.533828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.533830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.533920 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.534211 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.534214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.534216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.534385 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.534468 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.534700 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.534808 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 04:07:35.535171 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.527125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43867 (local address 127.0.0.1:38722)
0924 04:07:35.527219 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.527233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.527295 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.527526 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.527528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.527657 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.527770 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.527777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.528654 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.528656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.529380 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.529389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.529987 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.529991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.530107 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.530572 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.530587 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.531712 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.533477 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.533480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.533482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.533712 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.534050 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.534053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.534055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.534129 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.534469 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.534472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.534561 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.534641 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.535021 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 04:07:35.537076 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:35.538257 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.527897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.0.98.62:37619)
0924 04:07:35.528711 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.528714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.528723 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:07:35.528748 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.528750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.528781 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.528834 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.530115 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.530940 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.531731 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.532033 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.532117 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.534529 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.534532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.534535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.534560 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.536092 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.536533 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.536537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.536539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.536623 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.536883 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.536887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.536889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.537453 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.537677 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.537985 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.538107 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749}
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:07:35.540580 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } attempt: 3
I20260924 04:07:35.542074 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.533758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38190)
0924 04:07:35.533934 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.533946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.534024 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.534291 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.534294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.534445 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.534549 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.534556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.535542 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.535546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.536192 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.536202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.536674 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.536678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.536764 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.537117 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.537127 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.538650 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.540410 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.540413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.540416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.540611 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.540904 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.540907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.540909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.540994 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.541622 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.541625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.541715 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.541796 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.541927 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:07:35.542927 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.533801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.0.98.60:45501)
0924 04:07:35.533934 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.533938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.534100 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:07:35.534134 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.534136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.534177 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.534662 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.534746 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.535444 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.536305 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.536582 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.536672 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.538764 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.538767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.538769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.538788 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.540331 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.540714 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.540717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.540718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.540796 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.541089 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.541092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.541094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.541505 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.542588 (+ 1083us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.542658 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.542779 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:07:35 dist-test-slave-zcsk krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:07:35.546542 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.537542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38202)
0924 04:07:35.537646 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.537658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.537722 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.538190 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.538193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.538336 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.538409 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.538415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.539656 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.539659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.540374 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.540384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.541013 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.541016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.541085 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.541505 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.541516 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.542908 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.545110 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.545112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.545114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.545327 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.545587 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.545590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.545592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.545668 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.546104 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.546106 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.546182 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.546283 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.546403 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:07:35.547255 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746: Bootstrap starting.
I20260924 04:07:35.547323 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.537604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.0.98.60:45501)
0924 04:07:35.537924 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.537933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.537945 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:07:35.537975 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.537978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.538014 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.538096 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.538506 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.539544 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.540488 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.540843 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.540922 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.543026 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.543030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.543032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.543052 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.545008 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.545415 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.545418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.545419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.545486 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.545750 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.545753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.545755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.545982 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.546069 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.547043 (+ 974us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.547148 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260924 04:07:35.550002 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.539754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.0.98.61:43867)
0924 04:07:35.539874 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.539878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.539948 (+ 70us) server_negotiation.cc:407] Connection header received
0924 04:07:35.540076 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.540079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.540122 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.540202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.540900 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.541569 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.542627 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.542937 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.543001 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.546571 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.546575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.546577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.546603 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.548197 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.548647 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.548650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.548652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.548729 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.548997 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.549001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.549002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.549218 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.549395 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.549750 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.549861 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:07:35.550028 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.540917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45501 (local address 127.0.0.1:38212)
0924 04:07:35.541075 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:35.541087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:35.541148 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:35.541797 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:35.541800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:35.541958 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:35.542059 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.542066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.543122 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.543125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.543869 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:35.543878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.544488 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.544492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:35.544566 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.544936 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:35.544949 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:35.546341 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:35.548571 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.548574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.548577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.548817 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.549102 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:35.549104 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:35.549106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:35.549185 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:35.549586 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:35.549589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:35.549683 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:07:35.549756 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:07:35.549879 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 04:07:35.550364 22900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790: Bootstrap starting.
I20260924 04:07:35.550313 22825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:35.550627 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.541328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.98.60:45501)
0924 04:07:35.541588 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.541600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.541611 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:07:35.541636 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.541639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.541673 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.542171 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.542287 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.542990 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.544054 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.544355 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.544763 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.546492 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.546495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.546497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.546519 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.548452 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.548930 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.548933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.548935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.549009 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.549282 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.549285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.549287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.549478 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.550310 (+ 832us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.550382 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.550501 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
I20260924 04:07:35.551329 22825 log.cc:824] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:35.553025 22900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:35.553323 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8: Bootstrap starting.
I20260924 04:07:35.553632 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746: No bootstrap required, opened a new log
I20260924 04:07:35.553872 22900 log.cc:824] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:35.555212 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:35.555878 22750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:35.556365 22900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790: No bootstrap required, opened a new log
I20260924 04:07:35.556737 22750 log.cc:824] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:35.556598 22724 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f258770cb82d47adb66ead1ebd6571a8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49834
I20260924 04:07:35.557597 22825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } }
I20260924 04:07:35.557824 22825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f8d8bd27c74fb8bbe0b28ec145f746, State: Initialized, Role: FOLLOWER
I20260924 04:07:35.558154 22825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } }
I20260924 04:07:35.558779 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.558952 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8: No bootstrap required, opened a new log
I20260924 04:07:35.559093 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:35.559593 22825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:35.561888 22750 raft_consensus.cc:348] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } }
I20260924 04:07:35.562143 22750 raft_consensus.cc:729] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f258770cb82d47adb66ead1ebd6571a8, State: Initialized, Role: FOLLOWER
I20260924 04:07:35.562476 22750 consensus_queue.cc:260] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } }
I20260924 04:07:35.562397 22900 raft_consensus.cc:348] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } }
I20260924 04:07:35.562639 22900 raft_consensus.cc:729] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c738370cb74848089204f723361aa790, State: Initialized, Role: FOLLOWER
I20260924 04:07:35.562947 22900 consensus_queue.cc:260] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } }
I20260924 04:07:35.563732 22750 sys_catalog.cc:564] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:35.563817 22724 raft_consensus.cc:482] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:07:35.563961 22724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:35.564195 22900 sys_catalog.cc:564] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:35.565261 22724 raft_consensus.cc:504] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } }
I20260924 04:07:35.566897 22799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f258770cb82d47adb66ead1ebd6571a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746"
I20260924 04:07:35.567152 22799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:35.563377 22757 master_runner.cc:186] Error getting master registration for 127.0.98.62:37619: OK, instance_id { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" instance_seqno: 1790222855077728 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:07:35.568428 22757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:07:35.568523 22799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f258770cb82d47adb66ead1ebd6571a8 in term 1.
I20260924 04:07:35.566887 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f258770cb82d47adb66ead1ebd6571a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c738370cb74848089204f723361aa790"
I20260924 04:07:35.569038 22874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:07:35.569253 22945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:35.569314 22945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:35.569341 22699 leader_election.cc:302] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f8d8bd27c74fb8bbe0b28ec145f746, f258770cb82d47adb66ead1ebd6571a8; no voters:
I20260924 04:07:35.570331 22874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f258770cb82d47adb66ead1ebd6571a8 in term 1.
I20260924 04:07:35.566239 22724 leader_election.cc:288] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [CANDIDATE]: Term 1 election: Requested vote from peers 03f8d8bd27c74fb8bbe0b28ec145f746 (127.0.98.61:43867), c738370cb74848089204f723361aa790 (127.0.98.60:45501)
W20260924 04:07:35.571126 22921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:35.571199 22921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:35.571940 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.572057 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:35.571153 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.574317 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:35.575989 22682 master_runner.cc:263] No leader master found from master f258770cb82d47adb66ead1ebd6571a8
I20260924 04:07:35.576062 22682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:07:35.571395 22922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:07:35.576242 22922 raft_consensus.cc:686] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [term 1 LEADER]: Becoming Leader. State: Replica: f258770cb82d47adb66ead1ebd6571a8, State: Running, Role: LEADER
I20260924 04:07:35.576570 22922 consensus_queue.cc:237] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } }
W20260924 04:07:35.574048 22703 tablet.cc:2369] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:07:35.572808 22941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:35.577703 22941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:35.580614 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f258770cb82d47adb66ead1ebd6571a8. Latest consensus state: current_term: 1 leader_uuid: "f258770cb82d47adb66ead1ebd6571a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.580730 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:35.596225 22955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:35.597245 22955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:35.601212 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.581601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.98.60:45501)
0924 04:07:35.581709 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.581713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.589691 (+ 7978us) server_negotiation.cc:407] Connection header received
0924 04:07:35.589841 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.589844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.589888 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.589935 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.592706 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.593448 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.595623 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.595986 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.596061 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.597721 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.597725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.597727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.597754 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.599367 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.599828 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.599831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.599833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.599955 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.600223 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.600226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.600228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.600453 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.600561 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.600924 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.601042 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:07:35.604038 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.583148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.0.98.61:43867)
0924 04:07:35.583262 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.583266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.592149 (+ 8883us) server_negotiation.cc:407] Connection header received
0924 04:07:35.592313 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.592316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.592363 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.592408 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.593303 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.594023 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.595046 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.595340 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.595410 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.597591 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.597595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.597597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.597626 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.599300 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.602331 (+ 3031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.602335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.602337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.602425 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.602935 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.602939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.602941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.603174 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.603258 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.603727 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.603852 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:07:35.604667 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:35.580290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.0.98.62:37619)
0924 04:07:35.580403 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.580421 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.588198 (+ 7777us) server_negotiation.cc:407] Connection header received
0924 04:07:35.588383 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.588386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.588494 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.588547 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.590808 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.591592 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.594522 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:35.594868 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:35.594940 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:35.600331 (+ 5391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:35.600335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:35.600338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:35.600363 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:35.601876 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.602721 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.602725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.602726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.602811 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:35.603479 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:35.603483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:35.603484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:35.603694 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:35.603803 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:35.604388 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 04:07:35.604516 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:07:35.611796 22874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [term 1 FOLLOWER]: Refusing update from remote peer f258770cb82d47adb66ead1ebd6571a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:35.612325 22799 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [term 1 FOLLOWER]: Refusing update from remote peer f258770cb82d47adb66ead1ebd6571a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:35.612522 22922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:35.612881 22949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:35.616587 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f258770cb82d47adb66ead1ebd6571a8. Latest consensus state: current_term: 1 leader_uuid: "f258770cb82d47adb66ead1ebd6571a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.616724 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:35.616920 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f258770cb82d47adb66ead1ebd6571a8. Latest consensus state: current_term: 1 leader_uuid: "f258770cb82d47adb66ead1ebd6571a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.617053 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:35.623806 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f258770cb82d47adb66ead1ebd6571a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.623960 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:35.625293 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f258770cb82d47adb66ead1ebd6571a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.625427 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:35.627585 22922 sys_catalog.cc:455] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f258770cb82d47adb66ead1ebd6571a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.627699 22922 sys_catalog.cc:458] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:35.628697 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f258770cb82d47adb66ead1ebd6571a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f258770cb82d47adb66ead1ebd6571a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37619 } } peers { permanent_uuid: "03f8d8bd27c74fb8bbe0b28ec145f746" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43867 } } peers { permanent_uuid: "c738370cb74848089204f723361aa790" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45501 } } }
I20260924 04:07:35.628818 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:35.628790 22961 mvcc.cc:205] Tried to move back new op lower bound from 7332752816577433600 to 7332752816447709184. Current Snapshot: MvccSnapshot[applied={T|T < 7332752816577433600}]
I20260924 04:07:35.630028 22959 mvcc.cc:205] Tried to move back new op lower bound from 7332752816577433600 to 7332752816447709184. Current Snapshot: MvccSnapshot[applied={T|T < 7332752816577433600}]
I20260924 04:07:35.630503 22960 mvcc.cc:205] Tried to move back new op lower bound from 7332752816577433600 to 7332752816447709184. Current Snapshot: MvccSnapshot[applied={T|T < 7332752816577433600}]
I20260924 04:07:35.632473 22955 catalog_manager.cc:1347] Generated new cluster ID: cc2ba11f2f044cd5901936fac7fe82ec
I20260924 04:07:35.632543 22955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:35.649587 22955 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:35.650913 22955 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:35.676079 22955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f258770cb82d47adb66ead1ebd6571a8: Generated new TSK 0
I20260924 04:07:35.677012 22955 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:07:35.710762 22905 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.707677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.0.98.60:45501)
0924 04:07:35.707834 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.707839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.709652 (+ 1813us) server_negotiation.cc:407] Connection header received
0924 04:07:35.709865 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.709869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.709925 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.709979 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.710624 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38234: BlockingRecv error: recv got EOF from 127.0.0.1:38234 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:07:35.712193 22754 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.708434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.0.98.62:37619)
0924 04:07:35.708546 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.708551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.711244 (+ 2693us) server_negotiation.cc:407] Connection header received
0924 04:07:35.711420 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.711424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.711473 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.711523 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.712052 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49846: BlockingRecv error: recv got EOF from 127.0.0.1:49846 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 04:07:35.713713 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.709051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.0.98.61:43867)
0924 04:07:35.709176 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.709180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.712576 (+ 3396us) server_negotiation.cc:407] Connection header received
0924 04:07:35.712958 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.712962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.713016 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.713065 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.713591 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38746: BlockingRecv error: recv got EOF from 127.0.0.1:38746 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 04:07:35.719991 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.718360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.0.98.61:43867)
0924 04:07:35.718473 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.718478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.718937 (+ 459us) server_negotiation.cc:407] Connection header received
0924 04:07:35.719168 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.719171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.719254 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.719311 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.719849 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38760: BlockingRecv error: recv got EOF from 127.0.0.1:38760 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:07:35.725440 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.723741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.0.98.61:43867)
0924 04:07:35.723854 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.723858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.724457 (+ 599us) server_negotiation.cc:407] Connection header received
0924 04:07:35.724666 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.724669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.724723 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.724777 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.725329 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38774: BlockingRecv error: recv got EOF from 127.0.0.1:38774 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:07:35.731282 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.729718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.0.98.61:43867)
0924 04:07:35.729827 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.729831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.730346 (+ 515us) server_negotiation.cc:407] Connection header received
0924 04:07:35.730538 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.730541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.730595 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.730646 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.731176 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38778: BlockingRecv error: recv got EOF from 127.0.0.1:38778 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:07:35.739353 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.737718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.0.98.61:43867)
0924 04:07:35.737865 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.737869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.738432 (+ 563us) server_negotiation.cc:407] Connection header received
0924 04:07:35.738624 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.738628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.738679 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.738727 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.739238 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38786: BlockingRecv error: recv got EOF from 127.0.0.1:38786 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 04:07:35.752731 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.750880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.0.98.61:43867)
0924 04:07:35.751003 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.751007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.751632 (+ 625us) server_negotiation.cc:407] Connection header received
0924 04:07:35.751848 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.751851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.751930 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.751984 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.752608 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38800: BlockingRecv error: recv got EOF from 127.0.0.1:38800 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:07:35.772445 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.771203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.98.61:43867)
0924 04:07:35.771330 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.771335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.771571 (+ 236us) server_negotiation.cc:407] Connection header received
0924 04:07:35.771734 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.771738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.771789 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.771854 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.772324 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38814: BlockingRecv error: recv got EOF from 127.0.0.1:38814 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 04:07:35.807512 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.806503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.0.98.61:43867)
0924 04:07:35.806687 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.806692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.806727 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:07:35.806866 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.806869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.806917 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.806986 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.807389 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38818: BlockingRecv error: recv got EOF from 127.0.0.1:38818 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260924 04:07:35.874724 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:35.873608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.98.61:43867)
0924 04:07:35.873765 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:35.873770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:35.873858 (+ 88us) server_negotiation.cc:407] Connection header received
0924 04:07:35.874022 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:35.874025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:35.874087 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:35.874153 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:35.874621 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38826: BlockingRecv error: recv got EOF from 127.0.0.1:38826 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 04:07:36.005838 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:36.004835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.0.98.61:43867)
0924 04:07:36.004978 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:36.004982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:36.005039 (+ 57us) server_negotiation.cc:407] Connection header received
0924 04:07:36.005170 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:36.005172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:36.005220 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:36.005291 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:36.005718 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38840: BlockingRecv error: recv got EOF from 127.0.0.1:38840 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 04:07:36.265363 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:36.263985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.0.98.61:43867)
0924 04:07:36.264129 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:36.264134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:36.264246 (+ 112us) server_negotiation.cc:407] Connection header received
0924 04:07:36.264509 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:36.264513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:36.264568 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:36.264652 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:36.265222 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38856: BlockingRecv error: recv got EOF from 127.0.0.1:38856 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:07:36.570677 22945 catalog_manager.cc:1259] Loaded cluster ID: cc2ba11f2f044cd5901936fac7fe82ec
I20260924 04:07:36.570760 22945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790: loading cluster ID for follower catalog manager: success
I20260924 04:07:36.572536 22921 catalog_manager.cc:1259] Loaded cluster ID: cc2ba11f2f044cd5901936fac7fe82ec
I20260924 04:07:36.572616 22921 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746: loading cluster ID for follower catalog manager: success
I20260924 04:07:36.572964 22945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790: acquiring CA information for follower catalog manager: success
I20260924 04:07:36.574412 22945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c738370cb74848089204f723361aa790: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:07:36.574729 22921 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746: acquiring CA information for follower catalog manager: success
I20260924 04:07:36.575773 22921 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03f8d8bd27c74fb8bbe0b28ec145f746: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:07:36.783380 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:36.779502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.0.98.61:43867)
0924 04:07:36.779748 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:36.779752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:36.779767 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:07:36.779815 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:36.779817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:36.779970 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:36.780108 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:36.780349 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.780860 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.781762 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.782105 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.782199 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:36.782569 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:36.783019 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:36.783175 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:36.783202 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 04:07:36.783254 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260924 04:07:36.789920 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:36.786407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.0.98.60:45501)
0924 04:07:36.786645 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:36.786650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:36.786664 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:07:36.786747 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:36.786750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:36.786860 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:36.786966 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:36.787183 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.787692 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.788539 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.788759 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.788842 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:36.789109 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:36.789538 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:36.789642 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:36.789718 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 04:07:36.789772 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
I20260924 04:07:36.796772 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:36.792801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.0.98.60:45501)
0924 04:07:36.792897 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:36.792902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:36.793307 (+ 405us) server_negotiation.cc:407] Connection header received
0924 04:07:36.793385 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:36.793388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:36.793436 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:36.793499 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:36.793923 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.794472 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.795682 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.795987 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.796081 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:36.796294 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:36.796368 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:36.796429 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:36.796583 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:07:36.796636 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:07:36.796952 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:36.793112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.98.61:43867)
0924 04:07:36.793230 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:36.793234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:36.793568 (+ 334us) server_negotiation.cc:407] Connection header received
0924 04:07:36.793620 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:36.793622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:36.793661 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:36.793720 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:36.793995 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.794741 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.795810 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.796165 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.796241 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:36.796531 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:36.796602 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:36.796659 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:36.796784 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 04:07:36.796839 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:07:36.797223 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:36.792759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.0.98.62:37619)
0924 04:07:36.793091 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:36.793095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:36.793108 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:07:36.793214 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:36.793217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:36.793381 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:36.793489 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:36.793684 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.794490 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.795444 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:36.795688 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:36.795778 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:36.796080 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:36.796426 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:36.796505 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:36.797011 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 04:07:36.797070 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":69,"threads_started":1}
I20260924 04:07:36.806166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22682
I20260924 04:07:36.811682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22757
I20260924 04:07:36.816864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 22831
2026-09-24T04:07:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2974 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:07:36 dist-test-slave-zcsk krb5kdc[23049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:07:36 dist-test-slave-zcsk krb5kdc[23049](info): set up 2 sockets
Sep 24 04:07:36 dist-test-slave-zcsk krb5kdc[23049](info): commencing operation
krb5kdc: starting...
W20260924 04:07:37.776185 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/test-user.keytab.
Sep 24 04:07:37 dist-test-slave-zcsk krb5kdc[23049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:07:37.895236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35023,127.0.98.61:41935,127.0.98.60:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:38.004434 23062 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:07:38.004741 23062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:38.004777 23062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:38.004844 23062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:38.008857 23062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:38.008916 23062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:38.008942 23062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:38.008965 23062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:38.008989 23062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:38.015408 23062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35023,127.0.98.61:41935,127.0.98.60:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:38.016971 23062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:38.018258 23062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:38.027858 23070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:38.027870 23068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:38.027959 23062 server_base.cc:1034] running on GCE node
W20260924 04:07:38.027870 23067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:38.028770 23062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:38.029966 23062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:38.031150 23062 hybrid_clock.cc:648] HybridClock initialized: now 1790222858031139 us; error 31 us; skew 500 ppm
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:38.033921 23062 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:07:38.034868 23062 webserver.cc:460] Webserver started at http://127.0.98.62:34783/ using document root <none> and password file <none>
I20260924 04:07:38.035418 23062 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:38.035486 23062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:38.035688 23062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:38.037828 23062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/data/instance:
uuid: "343a274736924e148f468764cc5fb19a"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "95ad3ecda5c3f7fed2fc3b37e59e77d6"
server_key_iv: "ab645dd2c0b6a0663332b4142eb947d6"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.038393 23062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/wal/instance:
uuid: "343a274736924e148f468764cc5fb19a"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "95ad3ecda5c3f7fed2fc3b37e59e77d6"
server_key_iv: "ab645dd2c0b6a0663332b4142eb947d6"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.042147 23062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:07:38.044729 23077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:38.045991 23062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:07:38.046108 23062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/wal
uuid: "343a274736924e148f468764cc5fb19a"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "95ad3ecda5c3f7fed2fc3b37e59e77d6"
server_key_iv: "ab645dd2c0b6a0663332b4142eb947d6"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.046207 23062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:38.078671 23062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:38.082175 23062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:38.082383 23062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:38.089704 23062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35023
I20260924 04:07:38.089977 23129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35023 every 8 connection(s)
I20260924 04:07:38.090803 23062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:07:38.091583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 23062
I20260924 04:07:38.091698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-0/wal/instance
I20260924 04:07:38.092051 392 external_mini_cluster.cc:1428] Setting key bf8714e78fe9ddd4f8d6111dcfb45dfc
I20260924 04:07:38.094403 23130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:38.098477 23130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:38.115106 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.101082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.0.98.62:35023)
0924 04:07:38.101656 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.101676 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.102021 (+ 345us) server_negotiation.cc:407] Connection header received
0924 04:07:38.103152 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.103158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.103455 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.103641 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.104612 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.105924 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.106964 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.107558 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.107727 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.110106 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.110119 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.110131 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.110155 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.112128 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.112603 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.112608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.112610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.112726 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.113047 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.113050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.113051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.113528 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.113634 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.114032 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.114258 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"spinlock_wait_cycles":91392,"thread_start_us":174,"threads_started":1}
I20260924 04:07:38.115108 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.101082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35023 (local address 127.0.0.1:42926)
0924 04:07:38.101838 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.101891 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.102817 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.103692 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.103697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.104131 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.104414 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.104429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.106056 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.106059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.106840 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.106850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.107723 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.107725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.107846 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.108423 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.108440 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.109933 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.112240 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.112249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.112259 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.112498 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.112821 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.112824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.112825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.112955 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.113632 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.113638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.113789 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.113957 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.114352 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":78,"thread_start_us":174,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:07:38.117585 23130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:38.118361 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.118046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45376)
0924 04:07:38.118183 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.118280 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:07:38.118371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41935
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35023,127.0.98.61:41935,127.0.98.60:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:07:38.118592 23080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
I20260924 04:07:38.119483 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.119267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45380)
0924 04:07:38.119365 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.119415 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:07:38.119745 23130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41935: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
I20260924 04:07:38.169016 23130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } attempt: 1
I20260924 04:07:38.169981 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.169610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45384)
0924 04:07:38.169784 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.169850 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 04:07:38.170656 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.170458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45394)
0924 04:07:38.170557 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.170599 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:07:38.170849 23130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41935: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
W20260924 04:07:38.230144 23137 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:07:38.230480 23137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:38.230525 23137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:38.230589 23137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:38.234687 23137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:38.234763 23137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:38.234791 23137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:38.234813 23137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:38.234834 23137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:38.239014 23130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } attempt: 2
I20260924 04:07:38.239888 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.239553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45404)
0924 04:07:38.239699 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.239803 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:07:38.240617 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.240396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45414)
0924 04:07:38.240511 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.240558 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:07:38.240609 23137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35023,127.0.98.61:41935,127.0.98.60:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
W20260924 04:07:38.240732 23130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41935: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41935: connect: Connection refused (error 111)
I20260924 04:07:38.241832 23137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:38.242884 23137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:38.251729 23145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:38.251729 23142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:38.251722 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:38.252125 23137 server_base.cc:1034] running on GCE node
I20260924 04:07:38.252846 23137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:38.254091 23137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:38.255286 23137 hybrid_clock.cc:648] HybridClock initialized: now 1790222858255246 us; error 49 us; skew 500 ppm
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:38.258348 23137 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:07:38.259380 23137 webserver.cc:460] Webserver started at http://127.0.98.61:34061/ using document root <none> and password file <none>
I20260924 04:07:38.259999 23137 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:38.260080 23137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:38.260301 23137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:38.262442 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/data/instance:
uuid: "95cd87f201b24fd0baea46b5ef9d7263"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "9296ffeb899cea87997c99a7473057fd"
server_key_iv: "a62be4c931f34cd21d598653d9d42eb1"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.263015 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/wal/instance:
uuid: "95cd87f201b24fd0baea46b5ef9d7263"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "9296ffeb899cea87997c99a7473057fd"
server_key_iv: "a62be4c931f34cd21d598653d9d42eb1"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.267004 23137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:07:38.269826 23152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:38.271225 23137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:07:38.271399 23137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/wal
uuid: "95cd87f201b24fd0baea46b5ef9d7263"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "9296ffeb899cea87997c99a7473057fd"
server_key_iv: "a62be4c931f34cd21d598653d9d42eb1"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.271518 23137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:38.286391 23137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:38.289705 23137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:38.289919 23137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:38.296569 23137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41935
I20260924 04:07:38.296809 23204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41935 every 8 connection(s)
I20260924 04:07:38.297535 23137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:07:38.300295 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:38.304049 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:38.304808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 23137
I20260924 04:07:38.304931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-1/wal/instance
I20260924 04:07:38.305213 392 external_mini_cluster.cc:1428] Setting key b8bcd5c1a3b6c0adb356b38d6d1a7dd7
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:38.319530 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.306225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.0.98.62:35023)
0924 04:07:38.306373 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.306377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.307317 (+ 940us) server_negotiation.cc:407] Connection header received
0924 04:07:38.308438 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.308442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.308507 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.308581 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.309808 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.310418 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.311788 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.312083 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.312170 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.316113 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.316116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.316119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.316147 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.317480 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.317998 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.318002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.318004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.318088 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.318413 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.318416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.318418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.318605 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.318690 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.319223 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.319347 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:07:38.320400 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.306599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35023 (local address 127.0.0.1:42928)
0924 04:07:38.307126 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.307185 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.308211 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.308764 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.308770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.309181 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.309662 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.309675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.310556 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.310560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.311670 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.311680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.312236 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.312241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.312407 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.312965 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.312982 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.315853 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.317586 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.317591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.317605 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.317853 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.318172 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.318176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.318178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.318320 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.318702 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.318708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.318985 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.319157 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.319498 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":134,"threads_started":1}
I20260924 04:07:38.321880 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:07:38.330168 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.322373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45430)
0924 04:07:38.322476 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.322489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.322564 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.323065 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.323067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.323200 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.323291 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.323297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.324272 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.324275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.324868 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.324877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.325449 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.325452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.325523 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.325907 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.325920 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.327051 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.328670 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.328672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.328674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.328845 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.329132 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.329134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.329136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.329207 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.329636 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.329639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.329732 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.329864 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.329979 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 04:07:38.330221 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.322438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.0.98.61:41935)
0924 04:07:38.322732 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.322740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.322759 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:07:38.322793 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.322797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.322957 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.323059 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.323367 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.324177 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.324975 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.325343 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.325429 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.327160 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.327168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.327180 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.327203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.328580 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.328960 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.328966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.328968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.329058 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.329272 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.329274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.329275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.329522 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.329627 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.329963 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.330082 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:07:38.331444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
/tmp/dist-test-taskeFjGAy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39687
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35023,127.0.98.61:41935,127.0.98.60:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:07:38.331884 23130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } attempt: 3
I20260924 04:07:38.332078 23205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:38.333141 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.332873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48158)
0924 04:07:38.333012 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.333078 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 04:07:38.333321 23153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
I20260924 04:07:38.334349 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.334151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48162)
0924 04:07:38.334242 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.334289 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 04:07:38.334539 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:07:38.341117 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.332418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.0.98.61:41935)
0924 04:07:38.332542 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.332546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.332621 (+ 75us) server_negotiation.cc:407] Connection header received
0924 04:07:38.332737 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.332740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.332783 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.332863 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.333335 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.334047 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.335584 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.335875 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.336012 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.338109 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.338112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.338114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.338135 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.339577 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.340057 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.340060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.340061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.340133 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.340406 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.340409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.340410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.340579 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.340647 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.340917 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.341006 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:07:38.341207 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.332375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45444)
0924 04:07:38.332556 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.332569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.332660 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.332914 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.332918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.333100 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.333231 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.333238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.334168 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.334173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.335469 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.335483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.336024 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.336028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.336141 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.336574 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.336589 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.337966 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.339684 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.339688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.339691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.339969 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.340216 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.340219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.340221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.340314 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.340661 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.340664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.340776 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.340887 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.341041 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 04:07:38.342418 23130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:38.343160 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.342910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48174)
0924 04:07:38.343030 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.343084 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:07:38.343788 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.343550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48186)
0924 04:07:38.343682 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.343730 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 04:07:38.343981 23130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
I20260924 04:07:38.375191 23130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } attempt: 1
I20260924 04:07:38.376250 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.375793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48202)
0924 04:07:38.376030 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.376139 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 04:07:38.377146 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.376894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48212)
0924 04:07:38.377023 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.377079 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 04:07:38.377303 23130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
I20260924 04:07:38.383821 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } attempt: 1
I20260924 04:07:38.384671 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.384373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48216)
0924 04:07:38.384535 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.384596 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 04:07:38.385450 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.385217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48222)
0924 04:07:38.385329 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.385374 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:07:38.385679 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
W20260924 04:07:38.445761 23211 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:07:38.446084 23211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:07:38.446120 23211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:07:38.446184 23211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:07:38.450585 23211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:07:38.450661 23211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:07:38.450688 23211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:07:38.450711 23211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:07:38.450739 23211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:07:38.452507 23130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } attempt: 2
I20260924 04:07:38.453375 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.453052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48238)
0924 04:07:38.453225 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.453292 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 04:07:38.453831 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } attempt: 2
I20260924 04:07:38.454128 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.453894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48240)
0924 04:07:38.454003 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.454057 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:07:38.454286 23130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
I20260924 04:07:38.454478 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.454204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48246)
0924 04:07:38.454360 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.454406 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 04:07:38.455081 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.454891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48258)
0924 04:07:38.454981 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.455025 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 04:07:38.455195 23205 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39687: connect: Connection refused (error 111)
I20260924 04:07:38.457101 23211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35023,127.0.98.61:41935,127.0.98.60:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:07:38.458246 23211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:07:38.459312 23211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:07:38.468194 23218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:38.468196 23217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:07:38.468214 23220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:07:38.468425 23211 server_base.cc:1034] running on GCE node
I20260924 04:07:38.469012 23211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:07:38.470189 23211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:07:38.471369 23211 hybrid_clock.cc:648] HybridClock initialized: now 1790222858471352 us; error 29 us; skew 500 ppm
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:07:38.474200 23211 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:07:38.475173 23211 webserver.cc:460] Webserver started at http://127.0.98.60:44615/ using document root <none> and password file <none>
I20260924 04:07:38.475801 23211 fs_manager.cc:359] Metadata directory not provided
I20260924 04:07:38.475873 23211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:07:38.476104 23211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:07:38.478284 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/data/instance:
uuid: "82594427641846ada10d91a4374c252e"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "89de39de6d71ef958338b6b44be702fb"
server_key_iv: "15f829a42628ec941cda9695a609c3a6"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.478842 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/wal/instance:
uuid: "82594427641846ada10d91a4374c252e"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "89de39de6d71ef958338b6b44be702fb"
server_key_iv: "15f829a42628ec941cda9695a609c3a6"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.482678 23211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 04:07:38.485284 23227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:07:38.486699 23211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:07:38.486836 23211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/wal
uuid: "82594427641846ada10d91a4374c252e"
format_stamp: "Formatted at 2026-09-24 04:07:38 on dist-test-slave-zcsk"
server_key: "89de39de6d71ef958338b6b44be702fb"
server_key_iv: "15f829a42628ec941cda9695a609c3a6"
server_key_version: "encryptionkey@0"
I20260924 04:07:38.486944 23211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:07:38.500191 23211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:07:38.503314 23211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:07:38.503508 23211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:07:38.510461 23211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39687
I20260924 04:07:38.510715 23279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39687 every 8 connection(s)
I20260924 04:07:38.511474 23211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:07:38.514317 23280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:07:38.518366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu as pid 23211
I20260924 04:07:38.518138 23280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:38.518484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeFjGAy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222576786845-392-0/minicluster-data/master-2/wal/instance
I20260924 04:07:38.518738 392 external_mini_cluster.cc:1428] Setting key a3f413f4475bc5bfa9129c9e61cd28d1
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:07:38.531945 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.520023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.0.98.62:35023)
0924 04:07:38.520180 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.520184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.520720 (+ 536us) server_negotiation.cc:407] Connection header received
0924 04:07:38.521496 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.521500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.521549 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.521639 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.522672 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.523451 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.524848 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.525211 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.525314 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.528052 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.528056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.528059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.528092 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.529540 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.530075 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.530079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.530081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.530175 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.530655 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.530659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.530661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.530936 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.531062 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.531599 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.531735 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:07:38.532848 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.520123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.0.98.60:39687)
0924 04:07:38.520547 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.520561 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.520592 (+ 31us) server_negotiation.cc:407] Connection header received
0924 04:07:38.521284 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.521301 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.521532 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.521695 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.522520 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.523701 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.524842 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.525390 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.525510 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.528437 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.528443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.528454 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.528477 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.530113 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.530671 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.530677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.530678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.530755 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.531034 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.531038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.531040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.531330 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.531474 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.531745 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.532103 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":215,"server-negotiator.queue_time_us":298,"spinlock_wait_cycles":37888,"thread_start_us":143,"threads_started":1}
I20260924 04:07:38.532852 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.520122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35023 (local address 127.0.0.1:42942)
0924 04:07:38.520576 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.520623 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.521297 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.521692 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.521697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.522189 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.522546 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.522558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.523560 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.523563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.524733 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.524743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.525356 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.525358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.525508 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.526246 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.526265 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.527827 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.529650 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.529658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.529668 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.529962 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.530282 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.530285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.530287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.530490 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.531070 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.531076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.531350 (+ 274us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.531529 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.531875 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"spinlock_wait_cycles":896,"thread_start_us":112,"threads_started":1}
I20260924 04:07:38.534034 23280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:07:38.542797 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.534478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41935 (local address 127.0.0.1:45460)
0924 04:07:38.534590 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.534604 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.534676 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.534914 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.534916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.535075 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.535193 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.535199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.536042 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.536045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.536772 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.536780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.537366 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.537370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.537483 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.537941 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.537996 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.539423 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.541350 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.541354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.541356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.541554 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.541815 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.541817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.541819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.541907 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.542290 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.542293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.542396 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.542501 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.542634 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:07:38.542793 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.534583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.0.98.61:41935)
0924 04:07:38.534698 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.534702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.534713 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:07:38.534750 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.534752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.534790 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.534852 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.535326 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.535945 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.536874 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.537231 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.537327 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.539578 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.539582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.539586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.539618 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.541245 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.541652 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.541655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.541657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.541734 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.541999 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.542003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.542004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.542202 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.542276 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.542548 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.542641 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:07:38.543594 23280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:07:38.544312 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.535486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.0.98.62:35023)
0924 04:07:38.535589 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.535593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.535650 (+ 57us) server_negotiation.cc:407] Connection header received
0924 04:07:38.535754 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.535757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.535799 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.535860 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.536578 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.537208 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.538282 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.538487 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.538578 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.541182 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.541185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.541187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.541218 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.542645 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.543123 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.543127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.543129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.543213 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.543491 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.543495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.543497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.543704 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.543965 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.544056 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.544170 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:07:38.546339 23205 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } attempt: 3
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:07:38.553472 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.544099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48266)
0924 04:07:38.544305 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.544317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.544387 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.544621 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.544624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.544775 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.544863 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.544870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.545815 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.545818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.546385 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.546395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.547052 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.547055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.547141 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.548884 (+ 1743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.548898 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.550265 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.552101 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.552104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.552106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.552366 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.552597 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.552600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.552601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.552668 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.552983 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.552986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.553066 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.553164 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.553303 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260924 04:07:38.553426 23130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } attempt: 3
I20260924 04:07:38.553912 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.544313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.0.98.60:39687)
0924 04:07:38.544418 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.544423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.544436 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:07:38.544469 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.544472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.544509 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.544583 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.545056 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.545718 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.546523 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.546876 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.546957 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.550387 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.550391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.550394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.550417 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.552004 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.552449 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.552453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.552455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.552525 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.552739 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.552742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.552744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.552905 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.552984 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.553674 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.553784 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:07:38.554729 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.545487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.0.98.61:41935)
0924 04:07:38.545597 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.545600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.545611 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:07:38.545678 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.545680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.545719 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.545796 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.546323 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.547029 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.548079 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.548278 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.548350 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.551275 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.551278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.551282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.551307 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.552968 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.553395 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.553399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.553400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.553482 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.553912 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.553916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.553917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.554135 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.554394 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.554466 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.554577 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":8576}
I20260924 04:07:38.555281 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.546660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48278)
0924 04:07:38.546970 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.546983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.547055 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.547262 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.547265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.547409 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.547478 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.547483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.548856 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.548859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.549406 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.549416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.550012 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.550015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.550100 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.550488 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.550501 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.551866 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.553778 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.553781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.553783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.553996 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.554282 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.554285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.554287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.554366 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.554882 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.554885 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.554971 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.555036 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.555150 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260924 04:07:38.555796 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.546718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.0.98.60:39687)
0924 04:07:38.547050 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.547058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.547071 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:07:38.547135 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.547137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.547174 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.547239 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.547599 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.548751 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.549516 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.549869 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.549998 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.552127 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.552130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.552132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.552153 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.553676 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.554106 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.554108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.554110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.554186 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.554470 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.554474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.554475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.554774 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.555424 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.555509 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.555620 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"spinlock_wait_cycles":3840,"thread_start_us":110,"threads_started":1}
I20260924 04:07:38.558068 23280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e: Bootstrap starting.
I20260924 04:07:38.559221 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:07:38.559384 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263: Bootstrap starting.
I20260924 04:07:38.560931 23280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:38.561529 23205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:38.560556 23104 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "343a274736924e148f468764cc5fb19a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958
I20260924 04:07:38.561827 23280 log.cc:824] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:38.562242 23205 log.cc:824] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:38.564265 23280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e: No bootstrap required, opened a new log
I20260924 04:07:38.564441 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263: No bootstrap required, opened a new log
Sep 24 04:07:38 dist-test-slave-zcsk krb5kdc[23049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:07:38.568590 23205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } }
I20260924 04:07:38.568591 23280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } }
I20260924 04:07:38.568859 23280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82594427641846ada10d91a4374c252e, State: Initialized, Role: FOLLOWER
I20260924 04:07:38.568871 23205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cd87f201b24fd0baea46b5ef9d7263, State: Initialized, Role: FOLLOWER
I20260924 04:07:38.569213 23280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } }
I20260924 04:07:38.569214 23205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } }
I20260924 04:07:38.569480 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.553915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39687 (local address 127.0.0.1:48280)
0924 04:07:38.554403 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 04:07:38.554419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:07:38.554493 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:07:38.556055 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:07:38.556058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:07:38.556227 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:07:38.556323 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.556329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.557056 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.557059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.557802 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:07:38.557814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.563191 (+ 5377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.563196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:07:38.563297 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.563861 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:07:38.563878 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:07:38.565560 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:07:38.567701 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.567705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.567707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.568055 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.568388 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:07:38.568392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:07:38.568394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:07:38.568483 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:07:38.568954 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:07:38.568957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:07:38.569078 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 04:07:38.569161 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:07:38.569311 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401}
I20260924 04:07:38.569994 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.555419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.98.60:39687)
0924 04:07:38.555764 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.555768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.555780 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:07:38.555812 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.555815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.555856 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.556420 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.556486 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.556938 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.562681 (+ 5743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.563009 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.563560 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.565753 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.565758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.565761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.565789 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.567545 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.568195 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.568199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.568201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.568288 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.568595 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.568598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.568599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.568834 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.569670 (+ 836us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.569747 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.569862 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260924 04:07:38.570542 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.570673 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:38.570736 23205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:38.570739 23280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:38.570909 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.571033 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:38.573382 23130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a: Bootstrap starting.
W20260924 04:07:38.575948 23310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:38.576047 23310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:38.576822 23130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a: Neither blocks nor log segments found. Creating new log.
I20260924 04:07:38.574558 23211 master_runner.cc:186] Error getting master registration for 127.0.98.62:35023: OK, instance_id { permanent_uuid: "343a274736924e148f468764cc5fb19a" instance_seqno: 1790222858087659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:07:38.578329 23130 log.cc:824] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:07:38.578567 23211 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:07:38.575352 23137 master_runner.cc:186] Error getting master registration for 127.0.98.62:35023: OK, instance_id { permanent_uuid: "343a274736924e148f468764cc5fb19a" instance_seqno: 1790222858087659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:07:38.579313 23137 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:07:38.581171 23130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a: No bootstrap required, opened a new log
W20260924 04:07:38.581166 23311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:38.581240 23311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:38.584439 23130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } }
I20260924 04:07:38.584803 23130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a274736924e148f468764cc5fb19a, State: Initialized, Role: FOLLOWER
I20260924 04:07:38.585242 23130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } }
I20260924 04:07:38.585969 23315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.586212 23315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:38.587087 23130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:07:38.587172 23104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:07:38.587301 23104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:38.588797 23104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } }
I20260924 04:07:38.589915 23104 leader_election.cc:288] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [CANDIDATE]: Term 1 election: Requested vote from peers 95cd87f201b24fd0baea46b5ef9d7263 (127.0.98.61:41935), 82594427641846ada10d91a4374c252e (127.0.98.60:39687)
I20260924 04:07:38.591652 23179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "343a274736924e148f468764cc5fb19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95cd87f201b24fd0baea46b5ef9d7263"
I20260924 04:07:38.591943 23179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:38.593348 23179 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a274736924e148f468764cc5fb19a in term 1.
I20260924 04:07:38.594091 23254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "343a274736924e148f468764cc5fb19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "82594427641846ada10d91a4374c252e"
I20260924 04:07:38.594383 23254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:07:38.595734 23254 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a274736924e148f468764cc5fb19a in term 1.
I20260924 04:07:38.596319 23080 leader_election.cc:302] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a274736924e148f468764cc5fb19a, 95cd87f201b24fd0baea46b5ef9d7263; no voters:
I20260924 04:07:38.602643 23062 master_runner.cc:263] No leader master found from master 343a274736924e148f468764cc5fb19a
I20260924 04:07:38.602743 23062 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:07:38.602859 23315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [term 1 FOLLOWER]: Leader election won for term 1
W20260924 04:07:38.603668 23326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:07:38.603824 23326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:07:38.612361 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.596862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.0.98.62:35023)
0924 04:07:38.596978 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.596982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.603210 (+ 6228us) server_negotiation.cc:407] Connection header received
0924 04:07:38.603319 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.603322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.603372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.603445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.604467 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.605277 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.606642 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.606995 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.607065 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.608859 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.608863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.608866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.608894 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.610532 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.611018 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.611022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.611024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.611108 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.611375 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.611378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.611380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.611584 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.611702 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.612067 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.612195 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:07:38.612932 23315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [term 1 LEADER]: Becoming Leader. State: Replica: 343a274736924e148f468764cc5fb19a, State: Running, Role: LEADER
I20260924 04:07:38.613440 23315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } }
I20260924 04:07:38.616827 23315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 343a274736924e148f468764cc5fb19a. Latest consensus state: current_term: 1 leader_uuid: "343a274736924e148f468764cc5fb19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.617062 23315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:38.617604 23337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:07:38.618623 23337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:07:38.620321 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.597727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.98.61:41935)
0924 04:07:38.597844 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.597848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.604154 (+ 6306us) server_negotiation.cc:407] Connection header received
0924 04:07:38.604326 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.604330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.604380 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.604429 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.606876 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.607632 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.610648 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.610984 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.611064 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.613590 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.613594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.613597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.613626 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.615304 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.616621 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.616625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.616627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.616725 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.617028 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.617032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.617034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.617262 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.617379 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.619996 (+ 2617us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.620140 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:07:38.622267 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:38.598538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.98.60:39687)
0924 04:07:38.598657 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.598661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.605521 (+ 6860us) server_negotiation.cc:407] Connection header received
0924 04:07:38.605848 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.605851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.605900 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.605949 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.608787 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.609539 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.612710 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:38.613010 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:38.613086 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:38.617721 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:07:38.617726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:07:38.617729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:07:38.617761 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:07:38.619415 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.620815 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.620820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.620822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.620936 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:07:38.621244 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:07:38.621247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:07:38.621249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:07:38.621489 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:07:38.621609 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:38.621961 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 04:07:38.622088 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:07:38.631207 23254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [term 1 FOLLOWER]: Refusing update from remote peer 343a274736924e148f468764cc5fb19a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:38.634430 23335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:38.640053 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 343a274736924e148f468764cc5fb19a. Latest consensus state: current_term: 1 leader_uuid: "343a274736924e148f468764cc5fb19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.640220 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:38.641768 23179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [term 1 FOLLOWER]: Refusing update from remote peer 343a274736924e148f468764cc5fb19a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:07:38.645300 23335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:07:38.650525 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 343a274736924e148f468764cc5fb19a. Latest consensus state: current_term: 1 leader_uuid: "343a274736924e148f468764cc5fb19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.650709 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:38.659605 23335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "343a274736924e148f468764cc5fb19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.659768 23335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:38.661300 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "343a274736924e148f468764cc5fb19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.661464 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:38.661808 23338 mvcc.cc:205] Tried to move back new op lower bound from 7332752828943822848 to 7332752828885405696. Current Snapshot: MvccSnapshot[applied={T|T < 7332752828943822848}]
I20260924 04:07:38.665062 23315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "343a274736924e148f468764cc5fb19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.665221 23315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a [sys.catalog]: This master's current role is: LEADER
I20260924 04:07:38.669109 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "343a274736924e148f468764cc5fb19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a274736924e148f468764cc5fb19a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } peers { permanent_uuid: "95cd87f201b24fd0baea46b5ef9d7263" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41935 } } peers { permanent_uuid: "82594427641846ada10d91a4374c252e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39687 } } }
I20260924 04:07:38.669253 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:07:38.670526 23341 mvcc.cc:205] Tried to move back new op lower bound from 7332752828943822848 to 7332752828885405696. Current Snapshot: MvccSnapshot[applied={T|T < 7332752828943822848}]
I20260924 04:07:38.670822 23339 mvcc.cc:205] Tried to move back new op lower bound from 7332752828943822848 to 7332752828885405696. Current Snapshot: MvccSnapshot[applied={T|T < 7332752828943822848}]
I20260924 04:07:38.676828 23337 catalog_manager.cc:1347] Generated new cluster ID: bee05fc046ce4256949e44dd7f08e74a
I20260924 04:07:38.676940 23337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:07:38.717110 23337 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:07:38.718743 23337 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:07:38.740728 23337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 343a274736924e148f468764cc5fb19a: Generated new TSK 0
I20260924 04:07:38.741820 23337 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:07:38.773507 23284 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.770736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.0.98.60:39687)
0924 04:07:38.770914 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.770920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.772304 (+ 1384us) server_negotiation.cc:407] Connection header received
0924 04:07:38.772510 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.772514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.772582 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.772636 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.773352 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48304: BlockingRecv error: recv got EOF from 127.0.0.1:48304 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:07:38.775525 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.771557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.0.98.62:35023)
0924 04:07:38.771693 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.771703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.774335 (+ 2632us) server_negotiation.cc:407] Connection header received
0924 04:07:38.774528 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.774532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.774622 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.774677 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.775339 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42974: BlockingRecv error: recv got EOF from 127.0.0.1:42974 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:07:38.778339 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.776314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.0.98.61:41935)
0924 04:07:38.776441 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.776446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.777241 (+ 795us) server_negotiation.cc:407] Connection header received
0924 04:07:38.777432 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.777436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.777494 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.777557 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.778189 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45486: BlockingRecv error: recv got EOF from 127.0.0.1:45486 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:07:38.784080 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.782095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.98.61:41935)
0924 04:07:38.782217 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.782221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.782749 (+ 528us) server_negotiation.cc:407] Connection header received
0924 04:07:38.783006 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.783010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.783157 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.783245 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.783880 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45502: BlockingRecv error: recv got EOF from 127.0.0.1:45502 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 04:07:38.790364 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.788383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.0.98.61:41935)
0924 04:07:38.788528 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.788533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.789231 (+ 698us) server_negotiation.cc:407] Connection header received
0924 04:07:38.789484 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.789488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.789545 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.789596 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.790231 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45506: BlockingRecv error: recv got EOF from 127.0.0.1:45506 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:07:38.797161 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.795057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.0.98.61:41935)
0924 04:07:38.795208 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.795212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.796214 (+ 1002us) server_negotiation.cc:407] Connection header received
0924 04:07:38.796328 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.796332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.796399 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.796503 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.797024 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45522: BlockingRecv error: recv got EOF from 127.0.0.1:45522 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 04:07:38.804338 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.803176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.0.98.61:41935)
0924 04:07:38.803383 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.803388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.803451 (+ 63us) server_negotiation.cc:407] Connection header received
0924 04:07:38.803597 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.803600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.803659 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.803746 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.804213 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45528: BlockingRecv error: recv got EOF from 127.0.0.1:45528 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260924 04:07:38.815683 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.814488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.0.98.61:41935)
0924 04:07:38.814642 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.814647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.814775 (+ 128us) server_negotiation.cc:407] Connection header received
0924 04:07:38.814902 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.814905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.814968 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.815086 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.815559 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45532: BlockingRecv error: recv got EOF from 127.0.0.1:45532 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:07:38.834998 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.833811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.0.98.61:41935)
0924 04:07:38.833967 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.833972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.834066 (+ 94us) server_negotiation.cc:407] Connection header received
0924 04:07:38.834292 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.834296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.834354 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.834433 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.834881 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45544: BlockingRecv error: recv got EOF from 127.0.0.1:45544 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 04:07:38.870371 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.869213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.0.98.61:41935)
0924 04:07:38.869356 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.869361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.869488 (+ 127us) server_negotiation.cc:407] Connection header received
0924 04:07:38.869638 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.869642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.869701 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.869773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.870250 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45550: BlockingRecv error: recv got EOF from 127.0.0.1:45550 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:07:38.937661 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:38.936478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.0.98.61:41935)
0924 04:07:38.936637 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:38.936641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:38.936754 (+ 113us) server_negotiation.cc:407] Connection header received
0924 04:07:38.936908 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:38.936912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:38.936962 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:38.937032 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:38.937529 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45560: BlockingRecv error: recv got EOF from 127.0.0.1:45560 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 04:07:39.068787 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:39.067588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.0.98.61:41935)
0924 04:07:39.067724 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:39.067729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:39.067864 (+ 135us) server_negotiation.cc:407] Connection header received
0924 04:07:39.068062 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:39.068066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:39.068128 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:39.068201 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:39.068667 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45568: BlockingRecv error: recv got EOF from 127.0.0.1:45568 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 04:07:39.328074 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:07:39.326952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.0.98.61:41935)
0924 04:07:39.327118 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:39.327121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:39.327138 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:07:39.327272 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:39.327274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:39.327329 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:39.327411 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:07:39.327934 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45580: BlockingRecv error: recv got EOF from 127.0.0.1:45580 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 04:07:39.577401 23310 catalog_manager.cc:1259] Loaded cluster ID: bee05fc046ce4256949e44dd7f08e74a
I20260924 04:07:39.577489 23310 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263: loading cluster ID for follower catalog manager: success
I20260924 04:07:39.579228 23310 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263: acquiring CA information for follower catalog manager: success
I20260924 04:07:39.580487 23310 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95cd87f201b24fd0baea46b5ef9d7263: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:07:39.582341 23311 catalog_manager.cc:1259] Loaded cluster ID: bee05fc046ce4256949e44dd7f08e74a
I20260924 04:07:39.582401 23311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e: loading cluster ID for follower catalog manager: success
I20260924 04:07:39.583997 23311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e: acquiring CA information for follower catalog manager: success
I20260924 04:07:39.585188 23311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82594427641846ada10d91a4374c252e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:07:39.845983 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:39.842073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.0.98.61:41935)
0924 04:07:39.842302 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:39.842306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:39.842325 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:07:39.842506 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:39.842510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:39.842642 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:39.842747 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:39.843027 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.843600 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.844499 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.844733 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.844819 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:39.845162 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:39.845627 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:39.845719 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:39.845805 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 04:07:39.845857 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260924 04:07:39.852830 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:39.848875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.0.98.60:39687)
0924 04:07:39.849073 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:39.849077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:39.849090 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:07:39.849209 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:39.849211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:39.849330 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:39.849446 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:39.849747 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.850271 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.851218 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.851439 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.851516 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:39.851940 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:39.852382 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:39.852476 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:39.852629 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 04:07:39.852689 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260924 04:07:39.860353 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:39.855873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.98.61:41935)
0924 04:07:39.856006 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:39.856010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:39.856459 (+ 449us) server_negotiation.cc:407] Connection header received
0924 04:07:39.856603 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:39.856606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:39.856654 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:39.856712 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:39.857010 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.857778 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.859121 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.859460 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.859562 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:39.859860 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:39.859973 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:39.860032 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:39.860165 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:07:39.860219 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:07:39.860746 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:39.856015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.0.98.60:39687)
0924 04:07:39.856123 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:39.856127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:39.856478 (+ 351us) server_negotiation.cc:407] Connection header received
0924 04:07:39.856523 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:39.856525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:39.856584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:39.856708 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:39.856905 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.857713 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.859630 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.859856 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.859996 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:39.860284 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:39.860382 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:39.860445 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:39.860569 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 04:07:39.860623 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:07:39.861433 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:07:39.855873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.0.98.62:35023)
0924 04:07:39.856138 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 04:07:39.856143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:07:39.856361 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:07:39.856524 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:07:39.856527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:07:39.856702 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:07:39.856822 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:07:39.858041 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.858581 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.859576 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:07:39.859929 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:07:39.860023 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:07:39.860555 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:07:39.860940 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:07:39.861036 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:07:39.861173 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 04:07:39.861244 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260924 04:07:39.864405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 23062
I20260924 04:07:39.870082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 23137
I20260924 04:07:39.875293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeFjGAy/build/debug/bin/kudu with pid 23211
2026-09-24T04:07:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3057 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6032 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283080 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST